Oct 30 13:00:27.626235 ip-10-0-130-43 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 13:00:27.626245 ip-10-0-130-43 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 13:00:27.626252 ip-10-0-130-43 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 13:00:27.626553 ip-10-0-130-43 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 13:00:37.875631 ip-10-0-130-43 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 13:00:37.875649 ip-10-0-130-43 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 91e78db8c7684882ae2c468bfc3bbbe0 -- Oct 30 13:02:30.830556 ip-10-0-130-43 systemd[1]: Starting Kubernetes Kubelet... Oct 30 13:02:31.252049 ip-10-0-130-43 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:02:31.252049 ip-10-0-130-43 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 13:02:31.252049 ip-10-0-130-43 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:02:31.252049 ip-10-0-130-43 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:02:31.257870 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257782 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.257870 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257864 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.257870 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257869 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.257870 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257873 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.257870 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257877 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257882 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257886 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257891 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257895 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257899 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257903 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257906 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257911 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257915 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257919 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257926 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257933 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257937 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257941 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257945 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257949 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257953 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257957 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.258160 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257961 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257965 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257969 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257974 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257979 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257984 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257988 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257993 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.257997 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258000 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258005 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258010 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258014 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258019 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258024 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258029 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258034 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258038 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258042 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258047 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.258927 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258051 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258056 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258060 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258065 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258070 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258074 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258078 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258083 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258088 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258093 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258097 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258105 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258111 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258117 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258121 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258126 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258130 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258135 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258139 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258144 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.259753 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258148 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258153 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258158 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258162 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258167 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258171 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258176 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258180 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258184 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258189 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258194 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258198 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258202 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258207 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258211 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258215 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258220 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258224 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258228 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.260534 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258233 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258247 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258252 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258256 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258260 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258265 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258289 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258294 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258299 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258304 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258309 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258314 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258319 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258323 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258327 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258332 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258337 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258341 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258345 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.260990 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258350 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258355 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258361 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258365 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.258370 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259724 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259735 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259739 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259744 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259749 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259754 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259758 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259762 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259766 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259770 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259779 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259784 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259789 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259793 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259797 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.261456 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259802 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259806 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259810 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259815 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259819 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259824 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259828 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259832 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259837 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259842 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259846 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259850 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259855 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259859 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259863 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259868 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259873 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259877 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259881 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259886 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.261936 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259891 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259896 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259900 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259904 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259909 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259913 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259918 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259922 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259934 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259939 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259943 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259948 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259952 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259956 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259961 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259965 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259970 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259974 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259978 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.262519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259983 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259987 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259992 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.259996 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260000 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260006 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260011 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260015 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260022 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260028 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260032 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260037 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260041 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260045 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260050 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260054 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260059 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260063 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260067 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260072 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.263345 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260076 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260081 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260086 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260091 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260095 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260100 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260104 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260109 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260113 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260119 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260124 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260128 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260133 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260137 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260141 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260146 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260150 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260155 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260160 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.263989 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260164 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260169 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260173 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260177 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260182 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260186 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260191 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260195 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260199 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260203 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260208 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260215 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.260222 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263016 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263028 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263034 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263040 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263045 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263050 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.264464 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263055 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263059 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263063 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263068 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263073 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263077 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263090 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263095 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263100 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263104 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263109 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263113 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263117 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263122 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263126 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263131 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263136 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263140 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263144 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263149 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.264935 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263153 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263157 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263161 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263166 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263171 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263175 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263180 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263185 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263189 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263194 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263199 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263203 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263208 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263212 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263216 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263221 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263225 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263229 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263233 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263237 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.265620 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263242 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263247 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263251 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263255 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263260 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263264 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263283 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263288 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263292 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263296 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263300 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263305 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263310 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263314 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263320 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263328 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263334 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263339 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263345 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.266496 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263349 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263353 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263357 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263362 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263367 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263371 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263375 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263380 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263384 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263389 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263393 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263398 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263402 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263406 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263411 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263415 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263419 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263424 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263428 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263432 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.267382 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263436 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263441 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263445 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263449 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263454 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263458 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263462 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263467 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263471 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263475 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263479 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263484 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263489 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263493 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263498 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263502 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263507 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263512 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263516 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.267916 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263520 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.263528 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265170 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265186 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265196 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265203 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265211 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265216 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265223 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265231 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265236 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265241 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265247 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265252 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265257 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265262 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265281 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265286 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265291 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265296 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265307 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265312 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265319 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 13:02:31.268519 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265324 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265330 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265337 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265342 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265347 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265353 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265358 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265364 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265369 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265374 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265379 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265387 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265392 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265397 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265401 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265406 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265411 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265418 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265423 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265428 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265433 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265438 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265444 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265449 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265454 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 13:02:31.269201 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265459 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265464 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265469 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265474 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265482 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265487 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265491 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265496 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265503 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265509 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265514 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265519 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265524 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265529 2583 flags.go:64] FLAG: --help="false" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265534 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265540 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265545 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265550 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265556 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265561 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265566 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265571 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265575 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 13:02:31.269864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265580 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265585 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265591 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265596 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265601 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265605 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265610 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265615 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265619 2583 flags.go:64] FLAG: --lock-file="" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265624 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265628 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265633 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265642 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265647 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265652 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265657 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265663 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265668 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265673 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265678 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265690 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265695 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265702 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265706 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265711 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 13:02:31.270503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265717 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265721 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265726 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265731 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265736 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265748 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265753 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265758 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265763 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265767 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265772 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265777 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265782 2583 flags.go:64] FLAG: --port="10250" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265787 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265791 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05eed3293c1f792a6" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265796 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265801 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265806 2583 flags.go:64] FLAG: --register-node="true" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265811 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265817 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265822 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265827 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265832 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265838 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265843 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265848 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 13:02:31.271109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265853 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265857 2583 flags.go:64] FLAG: --runonce="false" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265862 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265867 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265872 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265877 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265882 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265888 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265893 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265898 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265903 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265908 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265913 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265917 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265922 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265927 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265933 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265941 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265946 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265951 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265958 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265963 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265968 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265972 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265977 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 13:02:31.271822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265982 2583 flags.go:64] FLAG: --v="2" Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265989 2583 flags.go:64] FLAG: --version="false" Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.265996 2583 flags.go:64] FLAG: --vmodule="" Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.266002 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.266008 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266226 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266233 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266238 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266243 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266248 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266252 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266256 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266261 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266265 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266284 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266290 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266294 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266299 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266303 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266307 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.272430 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266311 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266316 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266320 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266325 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266329 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266333 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266337 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266344 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266351 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266356 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266360 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266364 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266369 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266373 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266377 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266382 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266387 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266392 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266396 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.272931 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266401 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266405 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266409 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266413 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266418 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266422 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266427 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266431 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266436 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266441 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266446 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266450 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266454 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266458 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266463 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266467 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266472 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266476 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266481 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266485 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.273472 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266489 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266493 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266500 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266507 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266513 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266518 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266523 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266528 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266533 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266540 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266545 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266549 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266553 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266558 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266562 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266566 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266571 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266575 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266580 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266585 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.273942 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266589 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266593 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266599 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266603 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266608 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266613 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266617 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266622 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266626 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266630 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266635 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266639 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266643 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266648 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266655 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266659 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266664 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266669 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266673 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266677 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.274535 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266681 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266687 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266692 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266696 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266701 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266705 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266710 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266715 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266719 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266723 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266727 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266732 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.266775 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266851 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266859 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266864 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.275002 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266869 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266873 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266878 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266882 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266886 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266890 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266895 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266899 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266903 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266908 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266912 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266919 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266923 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266927 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266932 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266936 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266940 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266947 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266951 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266956 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.275401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266961 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266965 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266969 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266974 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266980 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266986 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.266991 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267000 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267004 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267009 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267013 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267018 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267030 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267035 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267039 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267043 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267047 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267052 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267056 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267061 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.275867 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267065 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267069 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267074 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267087 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267093 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267098 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267102 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267106 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267111 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267117 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267121 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267125 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267131 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267135 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267139 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267143 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267148 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267153 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267158 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267164 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.276360 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267169 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267173 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267178 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267182 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267187 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267192 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267196 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267201 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267205 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267209 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267213 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267218 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267222 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267227 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267231 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267237 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267241 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267246 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267250 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.276833 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267255 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267259 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267265 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267283 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267288 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267293 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267297 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267301 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267305 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267309 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267314 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267319 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267323 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267327 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267332 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267336 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267345 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267352 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267356 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.277288 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267362 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267366 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267370 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267375 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.267379 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.274751 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.274766 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.275490 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.275499 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275559 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275565 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275568 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275572 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275575 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275579 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275583 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275587 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275590 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275593 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.277728 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275596 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275599 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275602 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275605 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275607 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275610 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275613 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275616 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275635 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275638 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275642 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275645 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275649 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275654 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275657 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275659 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275662 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275665 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275668 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275671 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.278202 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275674 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275677 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275680 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275683 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275685 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275688 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275691 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275694 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275697 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275699 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275702 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275705 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275708 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275711 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275714 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275716 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275719 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275721 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275724 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275727 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.278709 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275729 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275732 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275735 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275737 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275741 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275743 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275747 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275749 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275752 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275755 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275758 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275761 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275764 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275767 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275779 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275783 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275786 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275789 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275792 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.279182 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275796 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275799 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275801 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275804 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275807 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275810 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275813 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275816 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275818 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275821 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275824 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275827 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275830 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275833 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275835 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275838 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275841 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275844 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275847 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.279636 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275849 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275853 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275855 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275859 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275861 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275864 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275866 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275869 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275872 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275875 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275878 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275880 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275883 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275886 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275888 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275891 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275894 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.280070 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.275896 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.275923 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276065 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276069 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276072 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276075 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276078 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276081 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276084 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276087 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276090 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276093 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276095 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276099 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276101 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276104 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.280477 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276106 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276109 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276112 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276115 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276118 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276120 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276123 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276127 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276131 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276133 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276136 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276139 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276142 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276145 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276148 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276150 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276153 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276156 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276158 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.280866 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276161 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276164 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276167 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276170 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276172 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276175 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276178 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276181 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276183 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276186 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276189 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276192 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276196 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276199 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276202 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276206 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276208 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276211 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276214 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.281384 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276216 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276219 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276222 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276224 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276227 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276229 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276232 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276234 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276237 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276240 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276242 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276245 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276248 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276251 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276254 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276257 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276259 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276262 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276265 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276283 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.281830 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276286 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276289 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276291 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276294 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276297 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276300 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276302 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276305 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276308 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276315 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276318 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276321 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276323 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276326 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276329 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276331 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276334 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276337 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276339 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.282317 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276342 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276345 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276348 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276351 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276353 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276357 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276360 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276362 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276365 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276368 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276371 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276374 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276376 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276379 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276382 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.276405 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:02:31.282754 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276454 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276460 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276463 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276466 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276469 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276472 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276475 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276483 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276486 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276489 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276492 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276494 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276497 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276499 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276503 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276506 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276509 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276512 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276514 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:02:31.283149 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276517 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276519 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276522 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276525 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276527 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276530 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276533 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276535 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276538 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276540 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276545 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276549 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276552 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276555 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276558 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276561 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276563 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276566 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276569 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:02:31.283601 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276571 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276574 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276582 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276585 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276588 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276590 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276593 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276595 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276598 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276601 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276604 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276607 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276609 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276612 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276616 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276620 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276624 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276627 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276630 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276633 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:02:31.284047 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276635 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276638 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276640 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276643 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276645 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276648 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276651 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276653 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276656 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276659 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276661 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276664 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276666 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276669 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276671 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276680 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276683 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276685 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276688 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276690 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:02:31.284615 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276696 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276699 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276702 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276705 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276707 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276710 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276712 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276715 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276718 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276721 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276723 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276726 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276729 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276731 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276734 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276736 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276739 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276741 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276744 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276747 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:02:31.285113 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276749 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276752 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276754 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276757 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276759 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276762 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276769 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:31.276772 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.276834 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.280414 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.281414 2583 server.go:1014] "Starting client certificate rotation" Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.281515 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.281549 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:02:31.285597 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.283854 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:02:31.289762 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.289735 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:02:31.302365 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.302343 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 13:02:31.306416 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.306399 2583 log.go:25] "Validated CRI v1 image API" Oct 30 13:02:31.307365 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.307337 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 13:02:31.312169 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.312141 2583 fs.go:135] Filesystem UUIDs: map[1d9ba08e-aa38-45eb-b302-67ae7a42efbb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bab503e6-8de1-4b83-90db-b046f47653d4:/dev/nvme0n1p4] Oct 30 13:02:31.312243 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.312168 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 13:02:31.317752 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.317632 2583 manager.go:217] Machine: {Timestamp:2026-10-30 13:02:31.316192432 +0000 UTC m=+0.382170591 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099541 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f31a88060c1433c340b413b36d489 SystemUUID:ec2f31a8-8060-c143-3c34-0b413b36d489 BootID:91e78db8-c768-4882-ae2c-468bfc3bbbe0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:f5:d2:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:f5:d2:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:57:18:db:1d:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 13:02:31.317752 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.317749 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 13:02:31.317888 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.317868 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 13:02:31.320871 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.320846 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 13:02:31.321018 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.320874 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 13:02:31.321060 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.321021 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 13:02:31.321060 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.321030 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 13:02:31.321119 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.321069 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:02:31.321119 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.321086 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:02:31.322352 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.322341 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 13:02:31.327094 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.327071 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 13:02:31.327239 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.327227 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 13:02:31.328728 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.328714 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 13:02:31.328789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.328733 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 13:02:31.328789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.328749 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 13:02:31.328789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.328758 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 13:02:31.328789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.328768 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 13:02:31.329131 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.329114 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:02:31.331779 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.331750 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 13:02:31.340305 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.340251 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:02:31.341761 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341743 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 13:02:31.341838 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341776 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 13:02:31.341968 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341958 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 13:02:31.342001 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341974 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 13:02:31.342001 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341981 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 13:02:31.342001 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341987 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 13:02:31.342001 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341993 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 13:02:31.342001 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.341999 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 13:02:31.342900 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.342891 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 13:02:31.342900 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.342899 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 13:02:31.342964 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.342907 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 13:02:31.342964 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.342913 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 13:02:31.342964 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.342923 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 13:02:31.342964 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.342933 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 13:02:31.344474 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.344462 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 13:02:31.344474 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.344474 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 13:02:31.348449 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.348432 2583 server.go:1263] "Started kubelet" Oct 30 13:02:31.348549 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.348515 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 13:02:31.348612 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.348533 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 13:02:31.348677 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.348640 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 13:02:31.349259 ip-10-0-130-43 systemd[1]: Started Kubernetes Kubelet. Oct 30 13:02:31.349661 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.349646 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 13:02:31.350011 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.349993 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 13:02:31.356248 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.356231 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 13:02:31.358306 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.358290 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 13:02:31.358973 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.358956 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 13:02:31.359203 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.359178 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.359748 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359729 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 13:02:31.359748 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359749 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 13:02:31.359858 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359729 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 13:02:31.359858 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359854 2583 factory.go:55] Registering systemd factory Oct 30 13:02:31.359924 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359871 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 13:02:31.359924 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359875 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 13:02:31.359924 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.359887 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 13:02:31.360123 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.360105 2583 factory.go:153] Registering CRI-O factory Oct 30 13:02:31.360123 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.360124 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 13:02:31.360254 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.360203 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 13:02:31.360254 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.360232 2583 factory.go:103] Registering Raw factory Oct 30 13:02:31.360254 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.360242 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 13:02:31.361446 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.361431 2583 manager.go:319] Starting recovery of all containers Oct 30 13:02:31.371589 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.371568 2583 manager.go:324] Recovery completed Oct 30 13:02:31.376400 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.376380 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 13:02:31.376479 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.376449 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6xgs" Oct 30 13:02:31.377420 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.377407 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.379110 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379087 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 13:02:31.379201 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.379089 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 13:02:31.379263 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379234 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.379263 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379261 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.379379 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379300 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.379862 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379846 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 13:02:31.379862 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379858 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 13:02:31.379976 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379877 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 13:02:31.379976 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.379877 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.380330 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.379091 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-43.ec2.internal.18e350490f9d8203 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-43.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-43.ec2.internal,},FirstTimestamp:2026-10-30 13:02:31.348404739 +0000 UTC m=+0.414382898,LastTimestamp:2026-10-30 13:02:31.348404739 +0000 UTC m=+0.414382898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-43.ec2.internal,}" Oct 30 13:02:31.382486 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.382466 2583 policy_none.go:50] "Start" Oct 30 13:02:31.382486 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.382490 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 13:02:31.382621 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.382504 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 13:02:31.382893 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.382874 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.382968 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.382906 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.382968 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.382920 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.384680 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.384668 2583 policy_none.go:44] "Start" Oct 30 13:02:31.396070 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.396048 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 13:02:31.396070 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.396047 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 13:02:31.396187 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.396047 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 13:02:31.422668 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.422636 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 13:02:31.425813 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.425801 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 13:02:31.425894 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.425826 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 13:02:31.425894 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.425834 2583 server.go:87] "Starting device plugin registration server" Oct 30 13:02:31.426065 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.426054 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 13:02:31.426109 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.426065 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 13:02:31.426212 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.426196 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 13:02:31.426303 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.426290 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 13:02:31.426303 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.426302 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 13:02:31.426807 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.426772 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 13:02:31.426893 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.426832 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.481238 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.481211 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 13:02:31.483380 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.482433 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 13:02:31.483380 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.482451 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 13:02:31.483380 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.482468 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 13:02:31.483380 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.482500 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 13:02:31.496408 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.496391 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:02:31.527215 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.527154 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.528925 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.528910 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.529018 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.528936 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.529018 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.528946 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.529018 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.528972 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.555443 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.555425 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.555510 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.555447 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-43.ec2.internal\": node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.583211 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.583190 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal"] Oct 30 13:02:31.583322 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.583240 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.584064 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.584048 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.584146 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.584079 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.584146 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.584103 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.586394 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.586379 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.586517 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.586505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.586569 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.586532 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.587095 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.587074 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.587171 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.587108 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.587171 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.587121 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.587171 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.587085 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.587328 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.587186 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.587328 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.587197 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.589945 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.589927 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.590034 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.589955 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:02:31.590643 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.590628 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:02:31.590718 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.590655 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:02:31.590718 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.590666 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:02:31.596897 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.596881 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.625030 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.625010 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-43.ec2.internal\" not found" node="ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.627373 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.627357 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-43.ec2.internal\" not found" node="ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.663085 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.663063 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.663162 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.663090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.663162 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.663107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c6cf73d04b1de3c177c6f664e9eb174-config\") pod \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" (UID: \"9c6cf73d04b1de3c177c6f664e9eb174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.696950 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.696926 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.763224 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.763201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c6cf73d04b1de3c177c6f664e9eb174-config\") pod \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" (UID: \"9c6cf73d04b1de3c177c6f664e9eb174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.763334 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.763227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.763334 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.763242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.763334 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.763329 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.763432 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.763333 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c6cf73d04b1de3c177c6f664e9eb174-config\") pod \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" (UID: \"9c6cf73d04b1de3c177c6f664e9eb174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.763432 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.763328 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.797184 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.797123 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.897993 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.897960 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:31.928143 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.928123 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.930286 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:31.930258 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:02:31.999037 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:31.999005 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:33.347953 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:33.347813 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:33.448347 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:33.448315 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:33.529801 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.529772 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 13:02:33.530200 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.529893 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 13:02:33.549062 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:33.549039 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:02:33.630823 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.630775 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:57:32 +0000 UTC" deadline="2028-08-12 05:38:44.941245088 +0000 UTC" Oct 30 13:02:33.630823 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.630823 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15640h36m11.310427397s" Oct 30 13:02:33.638002 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.637977 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:02:33.708131 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.708104 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:02:33.728107 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.728087 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:02:33.729570 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.729550 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:02:33.729755 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:33.729728 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146c4e2eccc94fc30a7e9ef0c46e8801.slice/crio-c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f WatchSource:0}: Error finding container c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f: Status 404 returned error can't find the container with id c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f Oct 30 13:02:33.734420 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.734404 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:02:33.740023 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:33.740002 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6cf73d04b1de3c177c6f664e9eb174.slice/crio-7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38 WatchSource:0}: Error finding container 7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38: Status 404 returned error can't find the container with id 7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38 Oct 30 13:02:33.742178 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.742159 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:02:33.747658 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.747639 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:02:33.977849 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:33.977766 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:02:34.574528 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.574501 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:02:34.578667 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.578642 2583 apiserver.go:52] "Watching apiserver" Oct 30 13:02:34.607168 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.607140 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 13:02:34.612538 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.612509 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 13:02:34.612865 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.612838 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-nsdcz","openshift-dns/node-resolver-gd79z","openshift-multus/multus-additional-cni-plugins-jd79k","openshift-ovn-kubernetes/ovnkube-node-w6wcv","openshift-network-operator/iptables-alerter-f696x","kube-system/konnectivity-agent-7zncw","kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal","openshift-image-registry/node-ca-8kg5l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal","openshift-multus/multus-qq4pk","openshift-multus/network-metrics-daemon-hjjpd","openshift-network-diagnostics/network-check-target-gfvdc","kube-system/global-pull-secret-syncer-kn246","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks"] Oct 30 13:02:34.615402 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.615385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.617580 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.617554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.617669 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.617641 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:34.620883 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.620863 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.623437 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.623235 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.625510 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.625494 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.627948 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.627930 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.630270 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.630249 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.632770 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.632748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.633251 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633203 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrlh9\"" Oct 30 13:02:34.633421 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633384 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbfrs\"" Oct 30 13:02:34.633714 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633689 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.633714 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633689 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 13:02:34.633867 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633785 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 13:02:34.633941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633877 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 13:02:34.633941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.633929 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.634108 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.634066 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 13:02:34.635864 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.635849 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.636538 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.636519 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 13:02:34.636816 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.636795 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 13:02:34.636908 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.636850 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.636908 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.636881 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.637054 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.637038 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 13:02:34.637151 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.637133 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxrfm\"" Oct 30 13:02:34.637286 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.637273 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 13:02:34.637343 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.637308 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.638178 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.637785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.638364 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.638338 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8jrt\"" Oct 30 13:02:34.640974 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.640950 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:34.641061 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.641007 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:34.643161 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.643083 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.643161 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.643102 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 13:02:34.643161 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.643132 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 13:02:34.643834 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.643816 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:34.644089 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.643996 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:34.645494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.645476 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 13:02:34.645677 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.645663 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r278v\"" Oct 30 13:02:34.645818 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.645800 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxh59\"" Oct 30 13:02:34.645900 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.645886 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 13:02:34.646041 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.646020 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 13:02:34.646225 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.646209 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.646298 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.646025 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8w9b7\"" Oct 30 13:02:34.646298 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.646213 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.646298 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.646093 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.646445 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.646435 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.648182 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.648161 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.648292 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.648237 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.648333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.648314 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vz5h4\"" Oct 30 13:02:34.648554 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.648532 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 13:02:34.652548 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.652367 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:02:34.652548 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.652418 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m26f9" Oct 30 13:02:34.660621 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.660481 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m26f9" Oct 30 13:02:34.709325 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.709298 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 13:02:34.720615 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjlz\" (UniqueName: \"kubernetes.io/projected/134089b3-8c7a-4f4b-8330-127fb60ba056-kube-api-access-qbjlz\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-serviceca\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysconfig\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-cnibin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-config\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134089b3-8c7a-4f4b-8330-127fb60ba056-host-slash\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720736 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-device-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.720769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720780 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-run\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-host\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-log-socket\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-bin\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720913 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-system-cni-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720937 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-system-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-netns\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.720982 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-var-lib-kubelet\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-tmp\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-node-log\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-bin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8jb\" (UniqueName: \"kubernetes.io/projected/fec11257-4c9d-48b9-9333-970360865ea7-kube-api-access-dh8jb\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.721097 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nzl\" (UniqueName: \"kubernetes.io/projected/e060d933-f69d-44fd-a416-6a9877c0cfed-kube-api-access-j6nzl\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6bs\" (UniqueName: \"kubernetes.io/projected/19c122c0-62b5-437d-91d5-3d519c7958f7-kube-api-access-bp6bs\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-dbus\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-systemd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-ovn\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-script-lib\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm8zj\" (UniqueName: \"kubernetes.io/projected/fbc1dd68-9c18-40cf-9d70-c812d51564fc-kube-api-access-nm8zj\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-agent-certs\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721344 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721376 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-kubelet-config\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-socket-dir-parent\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-multus\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721493 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-conf-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721516 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4f4\" (UniqueName: \"kubernetes.io/projected/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-kube-api-access-pw4f4\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.721789 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec11257-4c9d-48b9-9333-970360865ea7-tmp-dir\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-kubernetes\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-hostroot\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-kubelet\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-netns\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721885 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-multus-certs\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721937 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-etc-kubernetes\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721956 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fec11257-4c9d-48b9-9333-970360865ea7-hosts-file\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721971 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-registration-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.721986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-kubelet\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-systemd\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-netd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.722491 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722101 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-socket-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-sys\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722155 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbkp\" (UniqueName: \"kubernetes.io/projected/34b15028-7cbf-46f3-9ffd-d9a8369128af-kube-api-access-zfbkp\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovn-node-metrics-cert\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-cni-binary-copy\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-multus-daemon-config\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722331 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsqn\" (UniqueName: \"kubernetes.io/projected/48592583-022f-490d-b547-a4876a43da65-kube-api-access-dcsqn\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-lib-modules\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722373 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722395 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-systemd-units\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722420 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-etc-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722445 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-konnectivity-ca\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722471 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-sys-fs\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722498 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm7db\" (UniqueName: \"kubernetes.io/projected/d3cd14b5-9230-48f3-8f92-67425d7a0122-kube-api-access-cm7db\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-slash\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722534 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-host\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-cnibin\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722583 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-var-lib-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722659 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-conf\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-tuned\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722756 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722782 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-k8s-cni-cncf-io\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-env-overrides\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134089b3-8c7a-4f4b-8330-127fb60ba056-iptables-alerter-script\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-modprobe-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.723984 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.722947 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.734177 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.734146 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 13:02:34.734305 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.734276 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-prxcs\"" Oct 30 13:02:34.736354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.736283 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" event={"ID":"9c6cf73d04b1de3c177c6f664e9eb174","Type":"ContainerStarted","Data":"7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38"} Oct 30 13:02:34.737554 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.737532 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" event={"ID":"146c4e2eccc94fc30a7e9ef0c46e8801","Type":"ContainerStarted","Data":"c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f"} Oct 30 13:02:34.824145 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-systemd-units\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824167 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-etc-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824184 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-konnectivity-ca\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-sys-fs\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824213 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-systemd-units\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7db\" (UniqueName: \"kubernetes.io/projected/d3cd14b5-9230-48f3-8f92-67425d7a0122-kube-api-access-cm7db\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824243 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-etc-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824280 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-slash\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824297 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-host\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-cnibin\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.824333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824322 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-sys-fs\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-var-lib-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824365 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-slash\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-conf\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824402 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-tuned\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824445 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-k8s-cni-cncf-io\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824509 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-env-overrides\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134089b3-8c7a-4f4b-8330-127fb60ba056-iptables-alerter-script\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-modprobe-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjlz\" (UniqueName: \"kubernetes.io/projected/134089b3-8c7a-4f4b-8330-127fb60ba056-kube-api-access-qbjlz\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824711 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824756 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-serviceca\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysconfig\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824811 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-cnibin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.824941 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-config\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134089b3-8c7a-4f4b-8330-127fb60ba056-host-slash\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-device-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-konnectivity-ca\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824936 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-k8s-cni-cncf-io\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-run\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.824982 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-host\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825050 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-host\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825059 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-run\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-log-socket\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-bin\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-system-cni-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825162 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-system-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825170 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-conf\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-netns\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-var-lib-kubelet\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.825568 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-tmp\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825285 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-node-log\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-bin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8jb\" (UniqueName: \"kubernetes.io/projected/fec11257-4c9d-48b9-9333-970360865ea7-kube-api-access-dh8jb\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nzl\" (UniqueName: \"kubernetes.io/projected/e060d933-f69d-44fd-a416-6a9877c0cfed-kube-api-access-j6nzl\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825583 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6bs\" (UniqueName: \"kubernetes.io/projected/19c122c0-62b5-437d-91d5-3d519c7958f7-kube-api-access-bp6bs\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-dbus\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-systemd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825706 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-ovn\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-script-lib\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8zj\" (UniqueName: \"kubernetes.io/projected/fbc1dd68-9c18-40cf-9d70-c812d51564fc-kube-api-access-nm8zj\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-agent-certs\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-kubelet-config\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-socket-dir-parent\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826070 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-multus\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-conf-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4f4\" (UniqueName: \"kubernetes.io/projected/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-kube-api-access-pw4f4\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec11257-4c9d-48b9-9333-970360865ea7-tmp-dir\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-kubernetes\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.825995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-hostroot\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826014 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-kubelet\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-netns\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-multus-certs\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826401 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-kubernetes\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826408 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826450 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-hostroot\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-system-cni-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826565 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-system-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826608 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-bin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826631 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-kubelet\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.826799 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826684 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826717 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-netns\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826773 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-netns\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.826993 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-dbus\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827045 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-systemd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827087 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-ovn\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827066 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827506 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827564 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-kubelet-config\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:34.827657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827620 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-conf-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827813 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-script-lib\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-etc-kubernetes\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fec11257-4c9d-48b9-9333-970360865ea7-hosts-file\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-registration-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827910 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-etc-kubernetes\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-kubelet\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-systemd\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.827992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-registration-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.828084 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-netd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828103 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec11257-4c9d-48b9-9333-970360865ea7-tmp-dir\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828124 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fec11257-4c9d-48b9-9333-970360865ea7-hosts-file\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-socket-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-sys\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828236 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-multus-certs\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbkp\" (UniqueName: \"kubernetes.io/projected/34b15028-7cbf-46f3-9ffd-d9a8369128af-kube-api-access-zfbkp\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovn-node-metrics-cert\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.828334 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-cni-binary-copy\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-multus-daemon-config\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsqn\" (UniqueName: \"kubernetes.io/projected/48592583-022f-490d-b547-a4876a43da65-kube-api-access-dcsqn\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828436 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-cnibin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.828683 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.828452 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-lib-modules\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.829161 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.829131 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-systemd\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.829415 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.829325 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-tmp\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.829968 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.829944 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-sys\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.830074 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.830051 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-socket-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.830518 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.830495 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-tuned\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.830790 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.830745 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-cni-binary-copy\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.830790 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.830743 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.830912 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.830837 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.831086 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831067 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-multus\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.831579 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831553 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-config\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.831710 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831644 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-var-lib-kubelet\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.831776 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831708 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.831776 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134089b3-8c7a-4f4b-8330-127fb60ba056-host-slash\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.831921 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831796 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-device-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.831921 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831870 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-modprobe-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.832111 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.831945 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.832350 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.832320 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-env-overrides\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.833079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.832559 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.833079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.832629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:34.833079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.832744 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-serviceca\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.833298 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833134 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-socket-dir-parent\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.833298 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833259 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-lib-modules\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.833474 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833306 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysconfig\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.833474 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833409 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-host\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.833474 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.833454 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.333433694 +0000 UTC m=+3.151084666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:34.833883 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.833854 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:34.834017 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833894 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134089b3-8c7a-4f4b-8330-127fb60ba056-iptables-alerter-script\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.834017 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.833931 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.333897188 +0000 UTC m=+3.151548172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:34.834017 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833949 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-cnibin\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.834017 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833978 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-bin\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.834017 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.833993 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-node-log\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.834316 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834020 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.834316 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834040 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-var-lib-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.834316 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834059 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-log-socket\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.834316 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834247 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-netd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.834316 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834247 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-kubelet\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.834650 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834630 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-multus-daemon-config\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.834953 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.834918 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-agent-certs\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.837546 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.837523 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovn-node-metrics-cert\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.844120 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.844096 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7db\" (UniqueName: \"kubernetes.io/projected/d3cd14b5-9230-48f3-8f92-67425d7a0122-kube-api-access-cm7db\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:34.846942 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.846923 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8zj\" (UniqueName: \"kubernetes.io/projected/fbc1dd68-9c18-40cf-9d70-c812d51564fc-kube-api-access-nm8zj\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.847014 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.846942 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjlz\" (UniqueName: \"kubernetes.io/projected/134089b3-8c7a-4f4b-8330-127fb60ba056-kube-api-access-qbjlz\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.847014 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.846940 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4f4\" (UniqueName: \"kubernetes.io/projected/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-kube-api-access-pw4f4\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.847014 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.846926 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6bs\" (UniqueName: \"kubernetes.io/projected/19c122c0-62b5-437d-91d5-3d519c7958f7-kube-api-access-bp6bs\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.847295 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.847269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8jb\" (UniqueName: \"kubernetes.io/projected/fec11257-4c9d-48b9-9333-970360865ea7-kube-api-access-dh8jb\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.847456 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.847438 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:02:34.847529 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.847465 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:02:34.847529 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.847478 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:34.847529 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:34.847527 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.34751533 +0000 UTC m=+3.165166324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:34.849586 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.849544 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbkp\" (UniqueName: \"kubernetes.io/projected/34b15028-7cbf-46f3-9ffd-d9a8369128af-kube-api-access-zfbkp\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.850188 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.850165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsqn\" (UniqueName: \"kubernetes.io/projected/48592583-022f-490d-b547-a4876a43da65-kube-api-access-dcsqn\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.850317 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.850293 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nzl\" (UniqueName: \"kubernetes.io/projected/e060d933-f69d-44fd-a416-6a9877c0cfed-kube-api-access-j6nzl\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.929145 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.929113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qq4pk" Oct 30 13:02:34.940214 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:34.940180 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48592583_022f_490d_b547_a4876a43da65.slice/crio-c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a WatchSource:0}: Error finding container c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a: Status 404 returned error can't find the container with id c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a Oct 30 13:02:34.957384 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.957361 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:02:34.961229 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.961212 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:02:34.969002 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.968981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:34.971066 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:34.971018 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134089b3_8c7a_4f4b_8330_127fb60ba056.slice/crio-2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43 WatchSource:0}: Error finding container 2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43: Status 404 returned error can't find the container with id 2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43 Oct 30 13:02:34.974678 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.974656 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:34.981726 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.981293 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:02:34.983234 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.983166 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:02:34.985000 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:34.984976 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc1dd68_9c18_40cf_9d70_c812d51564fc.slice/crio-d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9 WatchSource:0}: Error finding container d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9: Status 404 returned error can't find the container with id d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9 Oct 30 13:02:34.987565 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.987463 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gd79z" Oct 30 13:02:34.994009 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:34.993981 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b15028_7cbf_46f3_9ffd_d9a8369128af.slice/crio-5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b WatchSource:0}: Error finding container 5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b: Status 404 returned error can't find the container with id 5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b Oct 30 13:02:34.994453 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:34.994439 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:02:34.997097 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:34.997070 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5276b8dc_f3cf_4d3b_a630_bd16aca9a373.slice/crio-23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350 WatchSource:0}: Error finding container 23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350: Status 404 returned error can't find the container with id 23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350 Oct 30 13:02:35.000659 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:35.000634 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec11257_4c9d_48b9_9333_970360865ea7.slice/crio-f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c WatchSource:0}: Error finding container f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c: Status 404 returned error can't find the container with id f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c Oct 30 13:02:35.006189 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:02:35.006158 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode060d933_f69d_44fd_a416_6a9877c0cfed.slice/crio-ea83e71b8f548130ed153dacc2d9fe4aab64359fc27373ed56d64889482cf0f7 WatchSource:0}: Error finding container ea83e71b8f548130ed153dacc2d9fe4aab64359fc27373ed56d64889482cf0f7: Status 404 returned error can't find the container with id ea83e71b8f548130ed153dacc2d9fe4aab64359fc27373ed56d64889482cf0f7 Oct 30 13:02:35.336678 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.336648 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:35.336678 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.336691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:35.336911 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.336840 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:35.336970 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.336909 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:36.336890744 +0000 UTC m=+4.154541726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:35.336970 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.336845 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:35.337079 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.337006 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:36.336993696 +0000 UTC m=+4.154644668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:35.438200 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.437548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:35.438200 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.437785 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:02:35.438200 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.437803 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:02:35.438200 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.437816 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:35.438200 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:35.437876 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:36.437859273 +0000 UTC m=+4.255510256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:35.663217 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.663046 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:34 +0000 UTC" deadline="2028-07-26 07:51:51.027688966 +0000 UTC" Oct 30 13:02:35.663217 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.663107 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15234h49m15.364586292s" Oct 30 13:02:35.749973 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.749937 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gd79z" event={"ID":"fec11257-4c9d-48b9-9333-970360865ea7","Type":"ContainerStarted","Data":"f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c"} Oct 30 13:02:35.755556 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.755527 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kg5l" event={"ID":"5276b8dc-f3cf-4d3b-a630-bd16aca9a373","Type":"ContainerStarted","Data":"23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350"} Oct 30 13:02:35.770633 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.770257 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" event={"ID":"34b15028-7cbf-46f3-9ffd-d9a8369128af","Type":"ContainerStarted","Data":"5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b"} Oct 30 13:02:35.783642 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.783593 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f696x" event={"ID":"134089b3-8c7a-4f4b-8330-127fb60ba056","Type":"ContainerStarted","Data":"2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43"} Oct 30 13:02:35.790252 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.790229 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qq4pk" event={"ID":"48592583-022f-490d-b547-a4876a43da65","Type":"ContainerStarted","Data":"c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a"} Oct 30 13:02:35.794617 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.793268 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9"} Oct 30 13:02:35.801438 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.801412 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7zncw" event={"ID":"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8","Type":"ContainerStarted","Data":"1262668397b838a46f4210fc7542e2926680afb5f468e17f9b365d8c8ce7558b"} Oct 30 13:02:35.805701 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.805675 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerStarted","Data":"8a7cb4e2fb986e6896bcae6c8351e7eb0e9536e18aea9cb565f9bb653865e781"} Oct 30 13:02:35.816773 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.816747 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" event={"ID":"9c6cf73d04b1de3c177c6f664e9eb174","Type":"ContainerStarted","Data":"a1e2081deaeb9dd5a8fd161b6eed4c4a9a32a67db41e8acc71b8cd8ae1eb01db"} Oct 30 13:02:35.836229 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.836201 2583 generic.go:356] "Generic (PLEG): container finished" podID="146c4e2eccc94fc30a7e9ef0c46e8801" containerID="4f5c6ceff6f29ec7c216d9ef79e526c941363e35e6532491570e786fb20fe62d" exitCode=0 Oct 30 13:02:35.836336 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.836249 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" event={"ID":"146c4e2eccc94fc30a7e9ef0c46e8801","Type":"ContainerDied","Data":"4f5c6ceff6f29ec7c216d9ef79e526c941363e35e6532491570e786fb20fe62d"} Oct 30 13:02:35.849810 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.849753 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" podStartSLOduration=2.84974252 podStartE2EDuration="2.84974252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:35.849496354 +0000 UTC m=+3.667147345" watchObservedRunningTime="2026-10-30 13:02:35.84974252 +0000 UTC m=+3.667393510" Oct 30 13:02:35.850881 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:35.850794 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" event={"ID":"e060d933-f69d-44fd-a416-6a9877c0cfed","Type":"ContainerStarted","Data":"ea83e71b8f548130ed153dacc2d9fe4aab64359fc27373ed56d64889482cf0f7"} Oct 30 13:02:36.346846 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.346508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:36.346846 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.346806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:36.347071 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.346952 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:36.347071 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.347013 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:38.346993958 +0000 UTC m=+6.164644945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:36.347455 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.347436 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:36.347554 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.347496 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:38.347481115 +0000 UTC m=+6.165132100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:36.448046 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.448012 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:36.448221 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.448207 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:02:36.448286 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.448225 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:02:36.448286 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.448237 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:36.448379 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.448291 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:38.448274624 +0000 UTC m=+6.265925600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:36.663778 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.663663 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:34 +0000 UTC" deadline="2028-05-17 12:57:23.306305862 +0000 UTC" Oct 30 13:02:36.663778 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.663724 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13559h54m46.642586465s" Oct 30 13:02:36.735103 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.734295 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:36.735103 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.734427 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:36.735103 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.734819 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:36.735103 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.734924 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:36.735103 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.734985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:36.735103 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:36.735055 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:36.860623 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:36.859922 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" event={"ID":"146c4e2eccc94fc30a7e9ef0c46e8801","Type":"ContainerStarted","Data":"418c7051dbc52136168c38e33b5873857ce9bb456c55c950a52a5329874697b0"} Oct 30 13:02:37.878630 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:37.877291 2583 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="85d0bc06002f9ac7622f0cce3198080952cf6c1db3bf38d7ff5e56fd8a2f4b31" exitCode=0 Oct 30 13:02:37.878630 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:37.878384 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerDied","Data":"85d0bc06002f9ac7622f0cce3198080952cf6c1db3bf38d7ff5e56fd8a2f4b31"} Oct 30 13:02:37.940950 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:37.940893 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" podStartSLOduration=4.940873427 podStartE2EDuration="4.940873427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:36.893066299 +0000 UTC m=+4.710717291" watchObservedRunningTime="2026-10-30 13:02:37.940873427 +0000 UTC m=+5.758524420" Oct 30 13:02:38.364574 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:38.363841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:38.364574 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:38.363899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:38.364574 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.364031 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:38.364574 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.364093 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:42.364076303 +0000 UTC m=+10.181727285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:38.364574 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.364485 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:38.364574 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.364533 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:42.364518308 +0000 UTC m=+10.182169289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:38.465243 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:38.464572 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:38.465243 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.464757 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:02:38.465243 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.464779 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:02:38.465243 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.464792 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:38.465243 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.464857 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:42.464838113 +0000 UTC m=+10.282489100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:38.733751 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:38.733674 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:38.733890 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.733805 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:38.734166 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:38.734151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:38.734243 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.734217 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:38.734281 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:38.734261 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:38.734333 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:38.734313 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:39.888817 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:39.888094 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7zncw" event={"ID":"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8","Type":"ContainerStarted","Data":"c80dd2dc6b394665e953c510338dae578748d517b45fd3a07159f49ef93881e7"} Oct 30 13:02:40.733376 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.733293 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:40.733521 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:40.733418 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:40.733807 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.733788 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:40.733982 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:40.733962 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:40.734047 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.734018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:40.734100 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:40.734087 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:40.893047 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.893011 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kg5l" event={"ID":"5276b8dc-f3cf-4d3b-a630-bd16aca9a373","Type":"ContainerStarted","Data":"670e8d193f497b78ebda9fe701df7cc25b7b70b12c926dc177d52a00890c1898"} Oct 30 13:02:40.896170 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.896145 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" event={"ID":"34b15028-7cbf-46f3-9ffd-d9a8369128af","Type":"ContainerStarted","Data":"4a95d2f1cd7ebe579b2c7ebf3a38ff4f0cf3a5bf1679fc1228750a3c84573361"} Oct 30 13:02:40.926426 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.926373 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7zncw" podStartSLOduration=5.167695806 podStartE2EDuration="8.926356369s" totalImagesPullingTime="3.758660563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:39.922538143 +0000 UTC m=+7.740189133" watchObservedRunningTime="2026-10-30 13:02:40.926356369 +0000 UTC m=+8.744007361" Oct 30 13:02:40.926561 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:40.926516 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8kg5l" podStartSLOduration=3.973738021 podStartE2EDuration="8.926505183s" totalImagesPullingTime="4.952767162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:40.925733548 +0000 UTC m=+8.743384539" watchObservedRunningTime="2026-10-30 13:02:40.926505183 +0000 UTC m=+8.744156174" Oct 30 13:02:41.941442 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:41.941413 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 13:02:42.396646 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.396613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:42.396808 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.396655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:42.396808 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.396756 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:42.396808 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.396801 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.396787705 +0000 UTC m=+18.214438686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:42.396945 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.396838 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:42.396945 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.396857 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.396850535 +0000 UTC m=+18.214501515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:42.497026 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.496989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:42.497254 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.497200 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:02:42.497254 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.497227 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:02:42.497254 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.497241 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:42.497416 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.497306 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.497288975 +0000 UTC m=+18.314939959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:42.508666 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.508225 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:42.510141 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.510104 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:42.678754 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.678653 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:02:41.941439261Z","UUID":"2dbc7eec-6e76-4c02-b250-3646fa4507f5","Handler":null,"Name":"","Endpoint":""} Oct 30 13:02:42.681395 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.681369 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 13:02:42.681395 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.681403 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 13:02:42.732650 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.732624 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:42.732768 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.732746 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:42.732839 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.732786 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:42.732890 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.732851 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:42.733106 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.732974 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:42.733106 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:42.733072 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:42.904288 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.904251 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gd79z" event={"ID":"fec11257-4c9d-48b9-9333-970360865ea7","Type":"ContainerStarted","Data":"835dd6659ad84a86fb2b6fa7fe676f9e9ae4f821ad79ec3e629ee371fec6d45f"} Oct 30 13:02:42.907945 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.907918 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" event={"ID":"34b15028-7cbf-46f3-9ffd-d9a8369128af","Type":"ContainerStarted","Data":"5ff952a47de231ae9c7a7a18c34565be713452ff35b0abd7fe412096e7d87a8e"} Oct 30 13:02:42.909637 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.909588 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qq4pk" event={"ID":"48592583-022f-490d-b547-a4876a43da65","Type":"ContainerStarted","Data":"302fa8837519c3249edac0e103b03e690c81195332ba8eeae9fa4fc699438c3e"} Oct 30 13:02:42.912997 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.912940 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerStarted","Data":"53772068dd80f0bc5f9ce428dfc54fb61b382e61c2a6a2b142b7f02911818739"} Oct 30 13:02:42.938908 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.938850 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gd79z" podStartSLOduration=3.364206328 podStartE2EDuration="10.93883201s" totalImagesPullingTime="7.574625682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.938365228 +0000 UTC m=+10.756016220" watchObservedRunningTime="2026-10-30 13:02:42.93883201 +0000 UTC m=+10.756483001" Oct 30 13:02:42.973259 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:42.973204 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qq4pk" podStartSLOduration=3.626832221 podStartE2EDuration="10.973187646s" totalImagesPullingTime="7.346355425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.972755153 +0000 UTC m=+10.790406167" watchObservedRunningTime="2026-10-30 13:02:42.973187646 +0000 UTC m=+10.790838637" Oct 30 13:02:43.918189 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:43.917576 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" event={"ID":"34b15028-7cbf-46f3-9ffd-d9a8369128af","Type":"ContainerStarted","Data":"54a3a91e1c789fc3d6a756cca708b4600503c574b62faecb9c33d8901b7e4f38"} Oct 30 13:02:43.920214 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:43.920172 2583 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="53772068dd80f0bc5f9ce428dfc54fb61b382e61c2a6a2b142b7f02911818739" exitCode=0 Oct 30 13:02:43.920214 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:43.920199 2583 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="83261853fa0fcb91dc49575a0af82a843be87bbbed0f96a4619055c4ccce06f3" exitCode=0 Oct 30 13:02:43.920389 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:43.920258 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerDied","Data":"53772068dd80f0bc5f9ce428dfc54fb61b382e61c2a6a2b142b7f02911818739"} Oct 30 13:02:43.920389 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:43.920304 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerDied","Data":"83261853fa0fcb91dc49575a0af82a843be87bbbed0f96a4619055c4ccce06f3"} Oct 30 13:02:43.988357 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:43.988297 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" podStartSLOduration=4.38616066 podStartE2EDuration="11.988282429s" totalImagesPullingTime="7.602121769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:43.952106285 +0000 UTC m=+11.769757269" watchObservedRunningTime="2026-10-30 13:02:43.988282429 +0000 UTC m=+11.805933423" Oct 30 13:02:44.732278 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.732052 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:44.732433 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.732087 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:44.732433 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:44.732351 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:44.732433 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:44.732408 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:44.732433 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.732123 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:44.732635 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:44.732466 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:44.924013 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.923968 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f696x" event={"ID":"134089b3-8c7a-4f4b-8330-127fb60ba056","Type":"ContainerStarted","Data":"a2de7f74335d36b554a9b006d3e03f5aa4555d5ea921dd0471498364b459613a"} Oct 30 13:02:44.926828 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.926802 2583 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="a7029f5c88655fce5df73670e7cc0cec52d53cc755daddbef5b9c28b9d6d2e2b" exitCode=0 Oct 30 13:02:44.926937 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.926880 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerDied","Data":"a7029f5c88655fce5df73670e7cc0cec52d53cc755daddbef5b9c28b9d6d2e2b"} Oct 30 13:02:44.957652 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:44.957574 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f696x" podStartSLOduration=5.278946097 podStartE2EDuration="12.95755577s" totalImagesPullingTime="7.678609673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:44.956996732 +0000 UTC m=+12.774647723" watchObservedRunningTime="2026-10-30 13:02:44.95755577 +0000 UTC m=+12.775206759" Oct 30 13:02:45.930885 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:45.930852 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" event={"ID":"e060d933-f69d-44fd-a416-6a9877c0cfed","Type":"ContainerStarted","Data":"a2db604dd9e548ea9dad743e8f491f45c097ee66392e744b6ebd4b29107263c2"} Oct 30 13:02:45.966181 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:45.966119 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" podStartSLOduration=4.108238882 podStartE2EDuration="13.966102194s" totalImagesPullingTime="9.857863312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:45.965584693 +0000 UTC m=+13.783235705" watchObservedRunningTime="2026-10-30 13:02:45.966102194 +0000 UTC m=+13.783753185" Oct 30 13:02:46.731695 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:46.731669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:46.731695 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:46.731689 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:46.731860 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:46.731794 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:46.731860 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:46.731843 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:46.731960 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:46.731888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:46.732010 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:46.731957 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:46.936152 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:46.936122 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerStarted","Data":"4651204352d563d906c39192f7b94e500112bcd199bfa997036f652dd6d8c0e3"} Oct 30 13:02:47.942494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:47.942294 2583 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="4651204352d563d906c39192f7b94e500112bcd199bfa997036f652dd6d8c0e3" exitCode=0 Oct 30 13:02:47.942494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:47.942351 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerDied","Data":"4651204352d563d906c39192f7b94e500112bcd199bfa997036f652dd6d8c0e3"} Oct 30 13:02:48.731319 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.731280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:48.731319 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.731297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:48.731539 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.731345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:48.731539 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:48.731442 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:48.731671 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:48.731551 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:48.731671 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:48.731638 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:48.841387 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.841348 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:48.841526 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.841474 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:02:48.842199 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.842150 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:02:48.947543 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.947514 2583 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="05149c82ff9699008e813c5c89a0ee1d3d5310cdb3cc6cb2a5eb5df1c02bef10" exitCode=0 Oct 30 13:02:48.947921 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:48.947594 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerDied","Data":"05149c82ff9699008e813c5c89a0ee1d3d5310cdb3cc6cb2a5eb5df1c02bef10"} Oct 30 13:02:49.952867 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.952619 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd79k" event={"ID":"19c122c0-62b5-437d-91d5-3d519c7958f7","Type":"ContainerStarted","Data":"ef356605c96c07e709c785d196c1946d9957922959e9a51df0d26f26dc509610"} Oct 30 13:02:49.955328 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.955304 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"c73892bff096a971952b840bb35c3673e1266990a2a47f7bdfbb0acf478bcad4"} Oct 30 13:02:49.955437 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.955369 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"938237577b680a7d3d8dfdae662b272257a4936d6c1cd824756c68201ae2202d"} Oct 30 13:02:49.955437 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.955398 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"a8e3d4830484f766fc131377361e9a13a1e8fa8d8059cf06c9675277cec7efce"} Oct 30 13:02:49.955437 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.955431 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"5cdb793f90da2a7d803925b67fc2be8760a05e37ffe0cf23619a298ca19a5838"} Oct 30 13:02:49.955521 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.955458 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"22f372ce8e306ced35b37427c2c72caf5fa215cff87441b628ec66159ee0f17f"} Oct 30 13:02:49.955521 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.955484 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"2def8b0614ce4704f383a5900b38855527def200a55edf7844b836ee7b145b87"} Oct 30 13:02:49.991756 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:49.990424 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jd79k" podStartSLOduration=7.902638395 podStartE2EDuration="17.990407031s" totalImagesPullingTime="10.087768636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:49.989242685 +0000 UTC m=+17.806893676" watchObservedRunningTime="2026-10-30 13:02:49.990407031 +0000 UTC m=+17.808058022" Oct 30 13:02:50.208072 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.207998 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.396062 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.396025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.457062 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.457029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:50.457062 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.457065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:50.457276 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.457155 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:50.457276 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.457158 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:50.457276 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.457202 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:06.457189883 +0000 UTC m=+34.274840864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:02:50.457276 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.457214 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:06.457208552 +0000 UTC m=+34.274859519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:02:50.557847 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.557764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:50.557989 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.557914 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:02:50.557989 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.557933 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:02:50.557989 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.557944 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:50.558076 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.557994 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:06.557979035 +0000 UTC m=+34.375630014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:02:50.560227 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.560200 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.694217 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.694178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.731071 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.731046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:50.731071 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.731066 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:50.731236 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:50.731203 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:50.731236 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.731222 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:50.731299 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.731271 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:50.731354 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:50.731334 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:51.961773 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:51.961692 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"1ef7be62f213827ca2992e0782071b1fbcc68d7bf1ec88c704fbf06d34c55209"} Oct 30 13:02:52.731999 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:52.731973 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:52.732141 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:52.732062 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:52.732141 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:52.732133 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:52.732222 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:52.732205 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:52.732290 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:52.732268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:52.732422 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:52.732401 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:53.971959 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:53.971875 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" event={"ID":"fbc1dd68-9c18-40cf-9d70-c812d51564fc","Type":"ContainerStarted","Data":"05630aecfa85d124eb7b1ebe2e02a4972f1d90b88972ce2e25dc172220b173a7"} Oct 30 13:02:53.972655 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:53.972082 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:53.987106 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:53.987083 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:54.044287 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.044244 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" podStartSLOduration=7.934061353 podStartE2EDuration="22.044232465s" totalImagesPullingTime="14.110171112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:54.043841027 +0000 UTC m=+21.861492033" watchObservedRunningTime="2026-10-30 13:02:54.044232465 +0000 UTC m=+21.861883455" Oct 30 13:02:54.223017 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.222932 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.228833 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.228813 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:54.353817 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.353778 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.731218 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.731187 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:54.731391 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.731188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:54.731391 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.731323 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:54.731391 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:54.731325 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:54.731391 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:54.731387 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:54.731572 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:54.731484 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:54.974331 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.974288 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:54.988532 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:54.988464 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:02:55.656870 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:55.656646 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfvdc"] Oct 30 13:02:55.656993 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:55.656895 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:55.657051 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:55.656988 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:55.659795 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:55.659767 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kn246"] Oct 30 13:02:55.659985 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:55.659866 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:55.659985 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:55.659969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:55.660496 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:55.660479 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hjjpd"] Oct 30 13:02:55.660577 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:55.660548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:55.660655 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:55.660641 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:56.214917 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:56.214880 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.357998 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:56.357969 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.731399 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:56.731372 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:56.731558 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:56.731484 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:57.495738 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:57.495707 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.720936 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:57.720905 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.731769 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:57.731749 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:57.731850 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:57.731769 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:57.731881 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:57.731865 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:57.731996 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:57.731978 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:58.719183 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:58.719152 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gd79z_fec11257-4c9d-48b9-9333-970360865ea7/dns-node-resolver/0.log" Oct 30 13:02:58.731245 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:58.731226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:02:58.731357 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:58.731317 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:02:58.901398 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:58.901362 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.058282 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:59.058247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.518851 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:59.518829 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kg5l_5276b8dc-f3cf-4d3b-a630-bd16aca9a373/node-ca/0.log" Oct 30 13:02:59.731765 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:59.731736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:02:59.732131 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:59.731746 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:02:59.732131 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:59.731833 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:02:59.732131 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:02:59.731914 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:02:59.790422 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:59.790351 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.986348 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:02:59.986310 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.126740 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:00.126645 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.458704 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:00.458631 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.732007 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:00.731981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:00.732363 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:00.732106 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:01.061945 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:01.061862 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:01.660424 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:01.660393 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:01.731150 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:01.731120 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:01.731309 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:01.731228 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:01.731309 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:01.731280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:01.731383 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:01.731368 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:01.863082 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:01.863048 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.164217 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:02.164186 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.311401 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:02.311364 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.445404 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:02.445324 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.582105 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:02.582076 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.716904 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:02.716817 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.731775 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:02.731752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:02.731913 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:02.731829 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:03.083114 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.083076 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.237539 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.237506 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.374549 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.374454 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.504275 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.504237 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.674280 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.674201 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.731412 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.731380 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:03.731562 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:03.731489 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:03.731562 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.731520 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:03.731654 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:03.731585 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:03.856449 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:03.856416 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.275938 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:04.275901 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.466042 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:04.466010 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.731535 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:04.731516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:04.731694 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:04.731632 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:05.071094 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:05.071017 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.201005 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:05.200968 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.656734 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:05.656706 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.731790 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:05.731762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:05.731937 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:05.731767 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:05.731999 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:05.731962 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:05.731999 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:05.731854 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:05.865505 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:05.865462 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.264016 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.263977 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.437277 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.437241 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.467108 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.467077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:06.467238 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.467114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:06.467238 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.467224 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:03:06.467311 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.467263 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:03:06.467311 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.467283 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret podName:d6aaafde-99f7-4cdb-9c95-381d18e5f1b1 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:38.467266928 +0000 UTC m=+66.284917913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret") pod "global-pull-secret-syncer-kn246" (UID: "d6aaafde-99f7-4cdb-9c95-381d18e5f1b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:03:06.467311 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.467301 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs podName:d3cd14b5-9230-48f3-8f92-67425d7a0122 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:38.467289581 +0000 UTC m=+66.284940565 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs") pod "network-metrics-daemon-hjjpd" (UID: "d3cd14b5-9230-48f3-8f92-67425d7a0122") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:03:06.567776 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.567700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:06.567912 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.567823 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:03:06.567912 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.567840 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:03:06.567912 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.567849 2583 projected.go:196] Error preparing data for projected volume kube-api-access-mbv2m for pod openshift-network-diagnostics/network-check-target-gfvdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:03:06.567912 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.567894 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m podName:89c9725b-8217-433e-8d03-d8a493c9f336 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:38.567880918 +0000 UTC m=+66.385531886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbv2m" (UniqueName: "kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m") pod "network-check-target-gfvdc" (UID: "89c9725b-8217-433e-8d03-d8a493c9f336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:03:06.573351 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.573323 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.731030 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.730997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:06.731416 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:06.731101 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:06.834091 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:06.834018 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.060722 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:07.060687 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.658804 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:07.658766 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.731564 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:07.731540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:07.731944 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:07.731647 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:07.731944 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:07.731733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:07.731944 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:07.731821 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:07.806002 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:07.805969 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.944329 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:07.944242 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:08.267314 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:08.267280 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:08.458375 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:08.458337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:08.731703 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:08.731677 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:08.732065 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:08.731780 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:09.058413 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:09.058332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.671964 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:09.671927 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.731221 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:09.731188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:09.731358 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:09.731294 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:09.731358 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:09.731344 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:09.731452 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:09.731435 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:09.860805 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:09.860776 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:10.263414 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:10.263374 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:10.465744 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:10.465712 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:10.731863 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:10.731840 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:10.731997 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:10.731944 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:10.852970 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:10.852924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:11.008340 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:11.008273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:11.171482 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:11.171446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:11.731017 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:11.730986 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:11.731180 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:11.730986 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:11.731180 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:11.731101 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:11.731180 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:11.731150 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:12.732007 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:12.731976 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:12.732378 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:12.732052 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:13.731739 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:13.731707 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:13.731889 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:13.731744 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:13.731889 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:13.731826 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:13.731967 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:13.731948 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:14.731834 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:14.731793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:14.732281 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:14.731925 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:15.731383 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:15.731355 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:15.731664 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:15.731468 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:15.731664 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:15.731529 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:15.731664 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:15.731641 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:16.731847 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:16.731813 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:16.732324 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:16.732290 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfvdc" podUID="89c9725b-8217-433e-8d03-d8a493c9f336" Oct 30 13:03:16.904408 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:16.904368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.056550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.056458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.201911 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.201874 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.335410 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.335312 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.571090 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.571060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.709160 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.709049 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.731822 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.731798 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:17.731947 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:17.731929 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn246" podUID="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" Oct 30 13:03:17.732234 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:17.731974 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:17.732234 ip-10-0-130-43 kubenswrapper[2583]: E1030 13:03:17.732060 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjpd" podUID="d3cd14b5-9230-48f3-8f92-67425d7a0122" Oct 30 13:03:18.003983 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.003957 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeReady" Oct 30 13:03:18.004139 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.004087 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 13:03:18.058811 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.058776 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dg69d"] Oct 30 13:03:18.061819 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.061800 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6zw7d"] Oct 30 13:03:18.061936 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.061920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.064929 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.064908 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.074415 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.074391 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 13:03:18.074512 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.074498 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 13:03:18.074552 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.074538 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tdfvc\"" Oct 30 13:03:18.079135 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.079114 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5zhld"] Oct 30 13:03:18.081577 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.081535 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 13:03:18.081850 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.081831 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.082869 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.082848 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dg69d"] Oct 30 13:03:18.083534 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.083512 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6zw7d"] Oct 30 13:03:18.100455 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.100384 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zhld"] Oct 30 13:03:18.100718 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.100698 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 13:03:18.100816 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.100790 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 13:03:18.100816 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.100798 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 13:03:18.100903 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.100827 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 13:03:18.148465 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.148434 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tcpr\" (UniqueName: \"kubernetes.io/projected/da949e5b-eb60-4823-b417-41067e8d0959-kube-api-access-5tcpr\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.148657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.148478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da949e5b-eb60-4823-b417-41067e8d0959-config-volume\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.148657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.148497 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da949e5b-eb60-4823-b417-41067e8d0959-tmp-dir\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.148657 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.148511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da949e5b-eb60-4823-b417-41067e8d0959-metrics-tls\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.176488 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.176458 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 13:03:18.176663 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.176592 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fx9p2\"" Oct 30 13:03:18.176738 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.176666 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 13:03:18.176738 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.176695 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwzrd\"" Oct 30 13:03:18.209859 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.209830 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86dc676d59-98t6x"] Oct 30 13:03:18.212755 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.212740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.231790 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.231744 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 13:03:18.231790 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.231777 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 13:03:18.231958 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.231808 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 13:03:18.231958 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.231854 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 13:03:18.232094 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.232073 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 13:03:18.233013 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.232992 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dc676d59-98t6x"] Oct 30 13:03:18.249700 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-crio-socket\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.249800 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249704 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-data-volume\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.249800 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e9b6ca-1d68-43eb-a809-24debe93ae5b-cert\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.249881 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tcpr\" (UniqueName: \"kubernetes.io/projected/da949e5b-eb60-4823-b417-41067e8d0959-kube-api-access-5tcpr\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.249881 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.249956 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da949e5b-eb60-4823-b417-41067e8d0959-config-volume\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.249956 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da949e5b-eb60-4823-b417-41067e8d0959-tmp-dir\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.249956 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.249956 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249934 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g884v\" (UniqueName: \"kubernetes.io/projected/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-api-access-g884v\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.249956 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblw7\" (UniqueName: \"kubernetes.io/projected/05e9b6ca-1d68-43eb-a809-24debe93ae5b-kube-api-access-qblw7\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.250186 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.249965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da949e5b-eb60-4823-b417-41067e8d0959-metrics-tls\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.250292 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.250274 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da949e5b-eb60-4823-b417-41067e8d0959-tmp-dir\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.250490 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.250469 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da949e5b-eb60-4823-b417-41067e8d0959-config-volume\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.253797 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.253777 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da949e5b-eb60-4823-b417-41067e8d0959-metrics-tls\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.266750 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.266698 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tcpr\" (UniqueName: \"kubernetes.io/projected/da949e5b-eb60-4823-b417-41067e8d0959-kube-api-access-5tcpr\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.274736 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.274713 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 13:03:18.274814 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.274769 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qckc2\"" Oct 30 13:03:18.274923 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.274906 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 13:03:18.351334 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.351483 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.351483 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g884v\" (UniqueName: \"kubernetes.io/projected/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-api-access-g884v\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.351483 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblw7\" (UniqueName: \"kubernetes.io/projected/05e9b6ca-1d68-43eb-a809-24debe93ae5b-kube-api-access-qblw7\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.351669 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351652 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.351723 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351715 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.351784 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.351913 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-crio-socket\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.351955 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-data-volume\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.351988 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.351958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e9b6ca-1d68-43eb-a809-24debe93ae5b-cert\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.352089 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-crio-socket\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.352089 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.352211 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352112 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.352211 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.352211 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.352354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352216 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.352354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.352279 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-data-volume\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.354137 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.354111 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.354437 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.354417 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e9b6ca-1d68-43eb-a809-24debe93ae5b-cert\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.370260 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.370234 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g884v\" (UniqueName: \"kubernetes.io/projected/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-api-access-g884v\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.370500 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.370483 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblw7\" (UniqueName: \"kubernetes.io/projected/05e9b6ca-1d68-43eb-a809-24debe93ae5b-kube-api-access-qblw7\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.376301 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.376285 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:18.383403 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.383378 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:18.394193 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.394170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:18.453220 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453193 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.453333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.453333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.453333 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.453494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.453494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.453494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.453371 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.454245 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.454224 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.454454 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.454439 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.454886 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.454866 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.455761 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.455715 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.458019 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.457995 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.459803 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.459781 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.470060 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.470021 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.525790 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.525725 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:18.706930 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.706898 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zhld"] Oct 30 13:03:18.709203 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.709172 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6zw7d"] Oct 30 13:03:18.710617 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.709975 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dg69d"] Oct 30 13:03:18.713208 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:18.713180 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2f8ea2_5dbd_4cf5_8be8_d725dbb7e3fc.slice/crio-4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41 WatchSource:0}: Error finding container 4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41: Status 404 returned error can't find the container with id 4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41 Oct 30 13:03:18.713921 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:18.713901 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda949e5b_eb60_4823_b417_41067e8d0959.slice/crio-00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a WatchSource:0}: Error finding container 00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a: Status 404 returned error can't find the container with id 00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a Oct 30 13:03:18.731113 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.731093 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:18.744133 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.744112 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:03:18.744133 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.744127 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\"" Oct 30 13:03:18.744439 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.744114 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:03:18.816547 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:18.816511 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dc676d59-98t6x"] Oct 30 13:03:18.820431 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:18.820402 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad840e4_950d_4761_a526_65c645e7de03.slice/crio-73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb WatchSource:0}: Error finding container 73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb: Status 404 returned error can't find the container with id 73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb Oct 30 13:03:19.027263 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.027205 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc676d59-98t6x" event={"ID":"5ad840e4-950d-4761-a526-65c645e7de03","Type":"ContainerStarted","Data":"73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb"} Oct 30 13:03:19.028710 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.028678 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zhld" event={"ID":"05e9b6ca-1d68-43eb-a809-24debe93ae5b","Type":"ContainerStarted","Data":"cabbb1b2249abb2037929bfff49c21ea7a86ba4aa2d5fee991f280b160b000b1"} Oct 30 13:03:19.030639 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.030610 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zw7d" event={"ID":"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc","Type":"ContainerStarted","Data":"7543a8c29d773dfc04aeca1f5d56ec23d09250b7cb05be610dabb84adfc7c895"} Oct 30 13:03:19.030744 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.030666 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zw7d" event={"ID":"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc","Type":"ContainerStarted","Data":"4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41"} Oct 30 13:03:19.031858 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.031831 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dg69d" event={"ID":"da949e5b-eb60-4823-b417-41067e8d0959","Type":"ContainerStarted","Data":"00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a"} Oct 30 13:03:19.732168 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.731673 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:19.732168 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.731723 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:19.755354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.755104 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\"" Oct 30 13:03:19.755354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.755198 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\"" Oct 30 13:03:19.755354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.755272 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:03:19.755354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:19.755315 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:03:20.038708 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.038623 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zw7d" event={"ID":"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc","Type":"ContainerStarted","Data":"804fb916e8fa22203ef576089ec5361ece774c96adf3ac62d92170edefb9537c"} Oct 30 13:03:20.354343 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.354253 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-857c87f86-l2ddg"] Oct 30 13:03:20.359350 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.359324 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.384114 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.384071 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:20.405293 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.405266 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 13:03:20.407516 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.407487 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857c87f86-l2ddg"] Oct 30 13:03:20.470028 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.469994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470215 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470215 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470215 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470352 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470403 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470445 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470400 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.470624 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.470514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.515686 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.515554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:20.571497 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571664 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571510 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571664 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571664 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571544 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571664 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571664 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571652 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571812 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.571812 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.571709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.572455 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.572434 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.573289 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.573125 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.573678 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.573650 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.574015 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.573989 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.574209 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.574179 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.579053 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.579029 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.582112 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.582092 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.595593 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.595557 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.672820 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.672745 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:20.935961 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.935825 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:20.993567 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:20.993541 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857c87f86-l2ddg"] Oct 30 13:03:21.001145 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:21.001111 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed238e7_3fbe_47d0_8748_eafbed34a837.slice/crio-cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d WatchSource:0}: Error finding container cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d: Status 404 returned error can't find the container with id cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d Oct 30 13:03:21.025149 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.024063 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx"] Oct 30 13:03:21.029348 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.028468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.044857 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.044237 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dg69d" event={"ID":"da949e5b-eb60-4823-b417-41067e8d0959","Type":"ContainerStarted","Data":"7992197fae580d0643b5bdf7dee7435911af94a60e5c9e846c25ba62854c0ff5"} Oct 30 13:03:21.044857 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.044283 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dg69d" event={"ID":"da949e5b-eb60-4823-b417-41067e8d0959","Type":"ContainerStarted","Data":"a61396df32093ec95b3a49520083b888ba2d25dc55cf1423604d1b1dfb39445e"} Oct 30 13:03:21.044857 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.044806 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:21.047181 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047037 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx"] Oct 30 13:03:21.047181 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047137 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 13:03:21.047351 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047183 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 13:03:21.047351 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047232 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 13:03:21.047580 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047543 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9dqpm\"" Oct 30 13:03:21.047580 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047549 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 13:03:21.047739 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.047644 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 13:03:21.048719 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.048684 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c87f86-l2ddg" event={"ID":"fed238e7-3fbe-47d0-8748-eafbed34a837","Type":"ContainerStarted","Data":"cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d"} Oct 30 13:03:21.086577 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.086513 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.175927 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.175891 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.176107 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.175981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.176107 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.176020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpr7\" (UniqueName: \"kubernetes.io/projected/78a085e4-85e9-466c-984c-e08ccc0b8862-kube-api-access-pgpr7\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.176107 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.176077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a085e4-85e9-466c-984c-e08ccc0b8862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.219704 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.219625 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.276882 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.276851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.276996 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.276947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.276996 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.276985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpr7\" (UniqueName: \"kubernetes.io/projected/78a085e4-85e9-466c-984c-e08ccc0b8862-kube-api-access-pgpr7\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.277095 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.277035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a085e4-85e9-466c-984c-e08ccc0b8862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.278710 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.278681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a085e4-85e9-466c-984c-e08ccc0b8862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.279663 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.279641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.281280 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.281261 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.294461 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.294435 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpr7\" (UniqueName: \"kubernetes.io/projected/78a085e4-85e9-466c-984c-e08ccc0b8862-kube-api-access-pgpr7\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.344393 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.344363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:21.583386 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.583350 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.643471 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.643411 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dg69d" podStartSLOduration=1.591762474 podStartE2EDuration="3.643383232s" totalImagesPullingTime="2.051620758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:21.097792816 +0000 UTC m=+48.915443807" watchObservedRunningTime="2026-10-30 13:03:21.643383232 +0000 UTC m=+49.461034225" Oct 30 13:03:21.643811 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.643786 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx"] Oct 30 13:03:21.654739 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:21.654698 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a085e4_85e9_466c_984c_e08ccc0b8862.slice/crio-7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3 WatchSource:0}: Error finding container 7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3: Status 404 returned error can't find the container with id 7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3 Oct 30 13:03:21.714315 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:21.714279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:22.054241 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:22.053784 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zhld" event={"ID":"05e9b6ca-1d68-43eb-a809-24debe93ae5b","Type":"ContainerStarted","Data":"bd4bba3518ec4252621e95c5999ed95318533b49afef742d63763e2418595651"} Oct 30 13:03:22.056344 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:22.056308 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zw7d" event={"ID":"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc","Type":"ContainerStarted","Data":"15c06a46e63a8863011de0d7a854857689b89a600cb39fd8649e7aeca236a538"} Oct 30 13:03:22.058554 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:22.058496 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" event={"ID":"78a085e4-85e9-466c-984c-e08ccc0b8862","Type":"ContainerStarted","Data":"7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3"} Oct 30 13:03:22.099498 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:22.094231 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5zhld" podStartSLOduration=0.842339046 podStartE2EDuration="4.09421336s" totalImagesPullingTime="3.251874314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:22.086983984 +0000 UTC m=+49.904634978" watchObservedRunningTime="2026-10-30 13:03:22.09421336 +0000 UTC m=+49.911864351" Oct 30 13:03:22.124304 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:22.124240 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6zw7d" podStartSLOduration=1.765216207 podStartE2EDuration="4.124221097s" totalImagesPullingTime="2.35900489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:03:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:22.122861574 +0000 UTC m=+49.940512564" watchObservedRunningTime="2026-10-30 13:03:22.124221097 +0000 UTC m=+49.941872089" Oct 30 13:03:22.975044 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:22.975016 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.064992 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.064542 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc676d59-98t6x" event={"ID":"5ad840e4-950d-4761-a526-65c645e7de03","Type":"ContainerStarted","Data":"657a2b2dd89c5454eee322cc091dcea656f8f0c457810707dbdca8666ade3b92"} Oct 30 13:03:23.067143 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.067119 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c87f86-l2ddg" event={"ID":"fed238e7-3fbe-47d0-8748-eafbed34a837","Type":"ContainerStarted","Data":"b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7"} Oct 30 13:03:23.068933 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.068906 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" event={"ID":"78a085e4-85e9-466c-984c-e08ccc0b8862","Type":"ContainerStarted","Data":"3aeb9fffe43879de42352a42c3b302f88debcc8ec0e1aef660e4feaac4c671ec"} Oct 30 13:03:23.105630 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.102728 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86dc676d59-98t6x" podStartSLOduration=1.425850688 podStartE2EDuration="5.102707792s" totalImagesPullingTime="3.676857104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.101250129 +0000 UTC m=+50.918901121" watchObservedRunningTime="2026-10-30 13:03:23.102707792 +0000 UTC m=+50.920358781" Oct 30 13:03:23.107687 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.107652 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.136666 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.136561 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-857c87f86-l2ddg" podStartSLOduration=1.2600340239999999 podStartE2EDuration="3.136547069s" totalImagesPullingTime="1.876513045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.135979971 +0000 UTC m=+50.953630961" watchObservedRunningTime="2026-10-30 13:03:23.136547069 +0000 UTC m=+50.954198058" Oct 30 13:03:23.896495 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:23.896457 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:24.076458 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:24.076419 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" event={"ID":"78a085e4-85e9-466c-984c-e08ccc0b8862","Type":"ContainerStarted","Data":"a16ea8c0c41a529ac81e04f6d2029d73719cf0f4dde625caeb2b36bd9cce366d"} Oct 30 13:03:24.111433 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:24.111377 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" podStartSLOduration=1.794053251 podStartE2EDuration="3.111363196s" totalImagesPullingTime="1.317309945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:24.110244606 +0000 UTC m=+51.927895620" watchObservedRunningTime="2026-10-30 13:03:24.111363196 +0000 UTC m=+51.929014185" Oct 30 13:03:25.367966 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:25.367935 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:25.674767 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:25.674688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:25.807713 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:25.807683 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.448958 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.448921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.594388 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.594353 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.620354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.620327 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n"] Oct 30 13:03:26.623640 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.623612 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.646368 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.646348 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 13:03:26.646368 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.646355 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 13:03:26.646530 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.646355 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hbwxh\"" Oct 30 13:03:26.649304 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.649272 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n"] Oct 30 13:03:26.653066 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.653048 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9"] Oct 30 13:03:26.657712 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.657250 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.673137 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.673118 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 13:03:26.673304 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.673286 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 13:03:26.680018 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.679999 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4gwc5\"" Oct 30 13:03:26.680099 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.680050 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 13:03:26.682118 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.682021 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qdws6"] Oct 30 13:03:26.685537 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.685517 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9"] Oct 30 13:03:26.685682 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.685666 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.720711 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.720643 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 13:03:26.727056 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.727035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.727167 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.727064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.727167 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.727112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwm5n\" (UniqueName: \"kubernetes.io/projected/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-kube-api-access-hwm5n\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.727246 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.727194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.746688 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.746664 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.773394 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.773370 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 13:03:26.773594 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.773581 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 13:03:26.773669 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.773584 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pbv95\"" Oct 30 13:03:26.828446 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828418 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27z9n\" (UniqueName: \"kubernetes.io/projected/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-kube-api-access-27z9n\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.828550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.828550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.828550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-metrics-client-ca\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.828550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828508 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.828731 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828576 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.828731 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.828793 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828755 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.828793 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828782 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-sys\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.828915 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-tls\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.828947 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828909 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbcd\" (UniqueName: \"kubernetes.io/projected/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-api-access-7jbcd\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.828947 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.829012 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.828969 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.829012 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.829079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-textfile\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.829079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdfaae46-aa7b-4311-8ea5-1f6716311a36-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.829150 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-root\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.829150 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-wtmp\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.829260 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwm5n\" (UniqueName: \"kubernetes.io/projected/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-kube-api-access-hwm5n\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.829810 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.829785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.831089 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.831071 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.831257 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.831238 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.845663 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.845641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwm5n\" (UniqueName: \"kubernetes.io/projected/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-kube-api-access-hwm5n\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.894275 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.894243 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.930346 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.930476 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.930476 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.930476 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-sys\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.930476 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930442 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-sys\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.930476 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930473 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-tls\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.930719 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930689 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbcd\" (UniqueName: \"kubernetes.io/projected/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-api-access-7jbcd\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.930771 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.930826 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-textfile\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.930876 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdfaae46-aa7b-4311-8ea5-1f6716311a36-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.930928 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-root\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.930928 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-wtmp\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931024 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.930970 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27z9n\" (UniqueName: \"kubernetes.io/projected/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-kube-api-access-27z9n\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931024 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931115 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931030 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-metrics-client-ca\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931115 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931054 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-root\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931115 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.931250 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931115 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.931422 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931388 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-textfile\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931489 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931407 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdfaae46-aa7b-4311-8ea5-1f6716311a36-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.931544 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-wtmp\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931746 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931726 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.931954 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931935 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.931989 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.931935 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-metrics-client-ca\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.933160 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.933135 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.933266 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.933146 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.933326 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.933298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-tls\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.933719 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.933702 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.938609 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.938583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:26.947672 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.947653 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27z9n\" (UniqueName: \"kubernetes.io/projected/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-kube-api-access-27z9n\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:26.947997 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.947976 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbcd\" (UniqueName: \"kubernetes.io/projected/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-api-access-7jbcd\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.973373 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.973308 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:26.993394 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:26.993372 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:27.000360 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.000329 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:27.012834 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:27.012800 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca8a3b3_0f1e_4bd2_8660_6d8add527517.slice/crio-0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92 WatchSource:0}: Error finding container 0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92: Status 404 returned error can't find the container with id 0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92 Oct 30 13:03:27.049503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.049479 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.086347 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.086319 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdws6" event={"ID":"2ca8a3b3-0f1e-4bd2-8660-6d8add527517","Type":"ContainerStarted","Data":"0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92"} Oct 30 13:03:27.187805 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.187756 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.246842 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.246809 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n"] Oct 30 13:03:27.250323 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:27.250299 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e86df5d_83ed_433e_9c0e_b5f59970dfe9.slice/crio-342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2 WatchSource:0}: Error finding container 342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2: Status 404 returned error can't find the container with id 342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2 Oct 30 13:03:27.276675 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.276568 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9"] Oct 30 13:03:27.278961 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:27.278939 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfaae46_aa7b_4311_8ea5_1f6716311a36.slice/crio-d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b WatchSource:0}: Error finding container d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b: Status 404 returned error can't find the container with id d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b Oct 30 13:03:27.325619 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.325567 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.470391 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.470346 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.714138 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.714056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.854695 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:27.854660 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.005065 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.005032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.092352 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.092310 2583 generic.go:356] "Generic (PLEG): container finished" podID="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" containerID="efc3fb82ddf97c403f382cb042d90c3f2d483907f48832175da49e1e33fbea41" exitCode=0 Oct 30 13:03:28.092498 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.092402 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdws6" event={"ID":"2ca8a3b3-0f1e-4bd2-8660-6d8add527517","Type":"ContainerDied","Data":"efc3fb82ddf97c403f382cb042d90c3f2d483907f48832175da49e1e33fbea41"} Oct 30 13:03:28.094808 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.094777 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" event={"ID":"bdfaae46-aa7b-4311-8ea5-1f6716311a36","Type":"ContainerStarted","Data":"d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b"} Oct 30 13:03:28.099405 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.099367 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" event={"ID":"6e86df5d-83ed-433e-9c0e-b5f59970dfe9","Type":"ContainerStarted","Data":"758f82eb745c76c68d66c74e7fd552b359e44df87ad9766a5069ab90d8d041dd"} Oct 30 13:03:28.099500 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.099436 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" event={"ID":"6e86df5d-83ed-433e-9c0e-b5f59970dfe9","Type":"ContainerStarted","Data":"3192cdacbf40e7dc2ad9f666cb51aa439c3539d3828520543a2d7101dd60ced8"} Oct 30 13:03:28.099500 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.099465 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" event={"ID":"6e86df5d-83ed-433e-9c0e-b5f59970dfe9","Type":"ContainerStarted","Data":"342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2"} Oct 30 13:03:28.149437 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.149292 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.526378 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.526337 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:28.526852 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.526442 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:28.531688 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:28.531666 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:29.104345 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.104306 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdws6" event={"ID":"2ca8a3b3-0f1e-4bd2-8660-6d8add527517","Type":"ContainerStarted","Data":"8e8f4d81cc6f82edddda1c5d14df5c2c5e5f74780e74092ee47a178d8aac68c9"} Oct 30 13:03:29.104545 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.104359 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdws6" event={"ID":"2ca8a3b3-0f1e-4bd2-8660-6d8add527517","Type":"ContainerStarted","Data":"2aae92ed8cb55b6669ecf32dc7adc2b9edd0dd1e961e40e4e339416f2d64f0ef"} Oct 30 13:03:29.109309 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.106791 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" event={"ID":"bdfaae46-aa7b-4311-8ea5-1f6716311a36","Type":"ContainerStarted","Data":"cf9fc993658d541d4df3d859a47395fe6c64ccc0a8569ad7e4b7425f1ccdf242"} Oct 30 13:03:29.109309 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.106840 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" event={"ID":"bdfaae46-aa7b-4311-8ea5-1f6716311a36","Type":"ContainerStarted","Data":"21cf64b1556edb76c62985fbb62d586e34dec717c6e73f17431a709f319390b7"} Oct 30 13:03:29.109309 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.106864 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" event={"ID":"bdfaae46-aa7b-4311-8ea5-1f6716311a36","Type":"ContainerStarted","Data":"03cb30d711007bd6aa2ab0827a6d08cc5045c0aa55492eee256e6335e2fe8fef"} Oct 30 13:03:29.111759 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.111728 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" event={"ID":"6e86df5d-83ed-433e-9c0e-b5f59970dfe9","Type":"ContainerStarted","Data":"aad4d70bc937499477e03136c7a3fa2193bdeafd2ebc14db42193d9cac35b9a7"} Oct 30 13:03:29.116928 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.116901 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:29.139702 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.139651 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qdws6" podStartSLOduration=2.203734243 podStartE2EDuration="3.139637597s" totalImagesPullingTime="855.38764ms" totalInitContainerRuntime="80.515714ms" isStatefulPod=true podCreationTimestamp="2026-10-30 13:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:29.13932223 +0000 UTC m=+56.956973220" watchObservedRunningTime="2026-10-30 13:03:29.139637597 +0000 UTC m=+56.957288587" Oct 30 13:03:29.215570 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.215487 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" podStartSLOduration=2.188488133 podStartE2EDuration="3.215473154s" totalImagesPullingTime="1.026985021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:29.214782603 +0000 UTC m=+57.032433593" watchObservedRunningTime="2026-10-30 13:03:29.215473154 +0000 UTC m=+57.033124144" Oct 30 13:03:29.249998 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.249952 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" podStartSLOduration=1.908110905 podStartE2EDuration="3.249937481s" totalImagesPullingTime="1.341826576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:29.24912387 +0000 UTC m=+57.066774860" watchObservedRunningTime="2026-10-30 13:03:29.249937481 +0000 UTC m=+57.067588471" Oct 30 13:03:29.949882 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:29.949843 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:30.081722 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:30.081673 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:30.672930 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:30.672893 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:30.673079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:30.672978 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:30.677553 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:30.677531 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:31.111132 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.111092 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79cdc67ddb-bx2c7"] Oct 30 13:03:31.114406 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.114388 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.122247 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.122224 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:31.127028 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.127011 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 13:03:31.137150 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.137081 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 13:03:31.137269 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.137244 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mk5rw\"" Oct 30 13:03:31.137269 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.137250 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 13:03:31.137431 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.137283 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2jm058lbepfmq\"" Oct 30 13:03:31.137431 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.137283 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 13:03:31.138555 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.138534 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79cdc67ddb-bx2c7"] Oct 30 13:03:31.171249 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.171354 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171293 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-tls\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.171476 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/187fc60f-7b2a-45dd-b5ff-13839099e07f-audit-log\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.171541 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npntf\" (UniqueName: \"kubernetes.io/projected/187fc60f-7b2a-45dd-b5ff-13839099e07f-kube-api-access-npntf\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.171541 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171527 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-metrics-server-audit-profiles\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.171694 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171680 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-client-certs\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.171730 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.171712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-client-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.218032 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.218008 2583 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dc676d59-98t6x"] Oct 30 13:03:31.273023 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.272990 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-tls\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273182 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273039 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/187fc60f-7b2a-45dd-b5ff-13839099e07f-audit-log\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273182 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273069 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npntf\" (UniqueName: \"kubernetes.io/projected/187fc60f-7b2a-45dd-b5ff-13839099e07f-kube-api-access-npntf\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273182 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273103 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-metrics-server-audit-profiles\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273347 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-client-certs\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273347 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-client-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273347 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273275 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.273571 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.273545 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/187fc60f-7b2a-45dd-b5ff-13839099e07f-audit-log\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.274080 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.274052 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.276628 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.274264 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-metrics-server-audit-profiles\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.276628 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.276152 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-client-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.276628 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.276297 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-tls\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.277494 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.277054 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-client-certs\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.290682 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.290662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npntf\" (UniqueName: \"kubernetes.io/projected/187fc60f-7b2a-45dd-b5ff-13839099e07f-kube-api-access-npntf\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.426073 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.425994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:31.482134 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.482107 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt"] Oct 30 13:03:31.486805 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.486779 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.489492 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.489419 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.502644 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.502265 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 13:03:31.502644 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.502545 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pjfzd\"" Oct 30 13:03:31.504294 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.504073 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt"] Oct 30 13:03:31.575523 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.575493 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkc7\" (UniqueName: \"kubernetes.io/projected/e85daeb3-1320-4a20-b831-0817f651f8e8-kube-api-access-hbkc7\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.575689 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.575548 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85daeb3-1320-4a20-b831-0817f651f8e8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.676900 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.676830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkc7\" (UniqueName: \"kubernetes.io/projected/e85daeb3-1320-4a20-b831-0817f651f8e8-kube-api-access-hbkc7\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.676900 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.676875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85daeb3-1320-4a20-b831-0817f651f8e8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.679539 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.679512 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85daeb3-1320-4a20-b831-0817f651f8e8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.693450 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.693429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkc7\" (UniqueName: \"kubernetes.io/projected/e85daeb3-1320-4a20-b831-0817f651f8e8-kube-api-access-hbkc7\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.701962 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.701935 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.731839 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.731805 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79cdc67ddb-bx2c7"] Oct 30 13:03:31.735691 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:31.735667 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187fc60f_7b2a_45dd_b5ff_13839099e07f.slice/crio-e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd WatchSource:0}: Error finding container e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd: Status 404 returned error can't find the container with id e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd Oct 30 13:03:31.804975 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.804938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:31.930359 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.930282 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-549bbd8b7d-lf666"] Oct 30 13:03:31.934754 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.934733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.954167 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.954147 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 13:03:31.965520 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.965499 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 13:03:31.966296 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.966277 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-549bbd8b7d-lf666"] Oct 30 13:03:31.981557 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981679 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981561 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-federate-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981679 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-serving-certs-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981783 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wfqr\" (UniqueName: \"kubernetes.io/projected/a56d5807-0ea6-48b5-8487-64330925498a-kube-api-access-6wfqr\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981783 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981762 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981877 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981804 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981922 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:31.981965 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:31.981936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-metrics-client-ca\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.046936 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.046903 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 13:03:32.047078 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.046966 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 13:03:32.047143 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.047126 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 13:03:32.047230 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.047198 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-467j9\"" Oct 30 13:03:32.047230 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.047219 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 13:03:32.071550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.071527 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:32.083361 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfqr\" (UniqueName: \"kubernetes.io/projected/a56d5807-0ea6-48b5-8487-64330925498a-kube-api-access-6wfqr\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083525 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083525 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083525 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083458 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083525 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083510 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-metrics-client-ca\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083742 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083742 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-federate-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.083742 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.083640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-serving-certs-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.084373 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.084318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-metrics-client-ca\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.084708 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.084688 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.084830 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.084790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-serving-certs-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.086239 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.086202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.086239 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.086202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.086396 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.086334 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.086471 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.086451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-federate-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.099668 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.099646 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfqr\" (UniqueName: \"kubernetes.io/projected/a56d5807-0ea6-48b5-8487-64330925498a-kube-api-access-6wfqr\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.109467 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.109442 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt"] Oct 30 13:03:32.113423 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:32.113400 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85daeb3_1320_4a20_b831_0817f651f8e8.slice/crio-6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d WatchSource:0}: Error finding container 6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d: Status 404 returned error can't find the container with id 6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d Oct 30 13:03:32.122955 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.122922 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" event={"ID":"187fc60f-7b2a-45dd-b5ff-13839099e07f","Type":"ContainerStarted","Data":"e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd"} Oct 30 13:03:32.126915 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.125235 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" event={"ID":"e85daeb3-1320-4a20-b831-0817f651f8e8","Type":"ContainerStarted","Data":"6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d"} Oct 30 13:03:32.250034 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.250007 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:32.580577 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.580443 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-549bbd8b7d-lf666"] Oct 30 13:03:32.585671 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.585641 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.588489 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:32.588459 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56d5807_0ea6_48b5_8487_64330925498a.slice/crio-25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21 WatchSource:0}: Error finding container 25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21: Status 404 returned error can't find the container with id 25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21 Oct 30 13:03:32.734079 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.734041 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.777443 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.777369 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d94544d6f-wrh5c"] Oct 30 13:03:32.780855 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.780824 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.805031 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.804384 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d94544d6f-wrh5c"] Oct 30 13:03:32.891330 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891237 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891330 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891289 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891422 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891474 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891502 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891550 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891761 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.891761 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.891706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.901253 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.901170 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.992277 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.992757 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.992483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.993172 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.993142 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.993283 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.993221 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.994001 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.993975 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.994228 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.994204 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.994295 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.994244 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.995483 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.995432 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:32.996037 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:32.996016 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:33.010884 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:33.010859 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:33.096040 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:33.095979 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:33.135903 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:33.135866 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" event={"ID":"a56d5807-0ea6-48b5-8487-64330925498a","Type":"ContainerStarted","Data":"25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21"} Oct 30 13:03:33.415487 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:33.415211 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d94544d6f-wrh5c"] Oct 30 13:03:33.422385 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:33.422354 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd22a5a_4f7f_41c1_b11c_d4ec7dc781a5.slice/crio-5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54 WatchSource:0}: Error finding container 5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54: Status 404 returned error can't find the container with id 5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54 Oct 30 13:03:33.724710 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:33.724674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:33.942853 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:33.942767 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.144289 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.144214 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" event={"ID":"187fc60f-7b2a-45dd-b5ff-13839099e07f","Type":"ContainerStarted","Data":"41570a2c0529e1996128131cdbe0fb1a22553351d9ad22d6386327205c0d6769"} Oct 30 13:03:34.148432 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.148400 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" event={"ID":"e85daeb3-1320-4a20-b831-0817f651f8e8","Type":"ContainerStarted","Data":"fce555ba90686d516722f2e0eacadadaec1e56e2f90d76c15a8cfc74ed068b1a"} Oct 30 13:03:34.148617 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.148579 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:34.150218 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.150193 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d94544d6f-wrh5c" event={"ID":"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5","Type":"ContainerStarted","Data":"3c80e9be7b7f5d5e27eb7c757432ef53857ef25039787bbdf344c7950304928a"} Oct 30 13:03:34.150324 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.150235 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d94544d6f-wrh5c" event={"ID":"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5","Type":"ContainerStarted","Data":"5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54"} Oct 30 13:03:34.154968 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.154456 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:34.179981 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.179930 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" podStartSLOduration=1.7226521460000002 podStartE2EDuration="3.179913741s" totalImagesPullingTime="1.457261595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:34.17894399 +0000 UTC m=+61.996594988" watchObservedRunningTime="2026-10-30 13:03:34.179913741 +0000 UTC m=+61.997564731" Oct 30 13:03:34.216905 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.216807 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d94544d6f-wrh5c" podStartSLOduration=2.216794146 podStartE2EDuration="2.216794146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:34.215621878 +0000 UTC m=+62.033272860" watchObservedRunningTime="2026-10-30 13:03:34.216794146 +0000 UTC m=+62.034445135" Oct 30 13:03:34.248808 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.248743 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" podStartSLOduration=1.677049607 podStartE2EDuration="3.248722186s" totalImagesPullingTime="1.571672579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:34.24740695 +0000 UTC m=+62.065057921" watchObservedRunningTime="2026-10-30 13:03:34.248722186 +0000 UTC m=+62.066373177" Oct 30 13:03:34.980874 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:34.980841 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.120860 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:35.120821 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.160692 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:35.160657 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" event={"ID":"a56d5807-0ea6-48b5-8487-64330925498a","Type":"ContainerStarted","Data":"4699cd38704072ea2a70c58f55312fb383233f23fcf95504cea43f91485a1b9c"} Oct 30 13:03:36.166798 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.166764 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" event={"ID":"a56d5807-0ea6-48b5-8487-64330925498a","Type":"ContainerStarted","Data":"e92bf2ef5bbb964cbcb9f3d5e0e111f5e33dbbeaac4345ca5e60391b48908538"} Oct 30 13:03:36.167263 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.166817 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" event={"ID":"a56d5807-0ea6-48b5-8487-64330925498a","Type":"ContainerStarted","Data":"0ac584f184a6df8f6ac8385d85137b0de29866fc95ceca678079e7c7279ebeae"} Oct 30 13:03:36.228878 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.228828 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" podStartSLOduration=2.402548359 podStartE2EDuration="5.228814081s" totalImagesPullingTime="2.826265722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:36.217493817 +0000 UTC m=+64.035144817" watchObservedRunningTime="2026-10-30 13:03:36.228814081 +0000 UTC m=+64.046465071" Oct 30 13:03:36.562542 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.562508 2583 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d94544d6f-wrh5c"] Oct 30 13:03:36.592258 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.592230 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d84dd8f56-zc48q"] Oct 30 13:03:36.595478 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.595458 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.615087 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.615063 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d84dd8f56-zc48q"] Oct 30 13:03:36.638933 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.638911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639052 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.638958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639108 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.639080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639150 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.639120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639150 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.639138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639245 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.639193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639294 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.639258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.639347 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.639296 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740191 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740345 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740345 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740262 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740345 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740345 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740513 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740513 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.740513 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.740383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.741493 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.741091 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.741493 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.741282 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.741493 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.741390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.741493 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.741458 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.741685 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.741529 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.743153 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.743128 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.743262 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.743244 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.758265 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.758239 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:36.906856 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:36.906768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:37.202412 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:37.202331 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d84dd8f56-zc48q"] Oct 30 13:03:37.208099 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:37.208066 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8fe315_cf83_4d6b_aeaf_25ef603b5ff4.slice/crio-a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd WatchSource:0}: Error finding container a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd: Status 404 returned error can't find the container with id a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd Oct 30 13:03:38.176879 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.176842 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d84dd8f56-zc48q" event={"ID":"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4","Type":"ContainerStarted","Data":"0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794"} Oct 30 13:03:38.177055 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.176899 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d84dd8f56-zc48q" event={"ID":"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4","Type":"ContainerStarted","Data":"a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd"} Oct 30 13:03:38.213527 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.213468 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d84dd8f56-zc48q" podStartSLOduration=2.213454139 podStartE2EDuration="2.213454139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:38.212145097 +0000 UTC m=+66.029796086" watchObservedRunningTime="2026-10-30 13:03:38.213454139 +0000 UTC m=+66.031105166" Oct 30 13:03:38.290640 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.290593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:38.421516 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.421482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:38.564786 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.564751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:38.564938 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.564832 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:38.576467 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.576437 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:03:38.576467 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.576456 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:03:38.588074 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.588052 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:38.588138 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.588078 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:38.665441 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.665406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:38.665581 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.665471 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\"" Oct 30 13:03:38.667224 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.667207 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\"" Oct 30 13:03:38.674371 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.674339 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:38.676535 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.676521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:38.676684 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.676672 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:03:38.697934 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.697908 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:03:38.709463 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.709438 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:38.871358 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.871281 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\"" Oct 30 13:03:38.880487 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.880466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:38.973728 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.973696 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kn246"] Oct 30 13:03:38.977420 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:38.977395 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6aaafde_99f7_4cdb_9c95_381d18e5f1b1.slice/crio-48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8 WatchSource:0}: Error finding container 48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8: Status 404 returned error can't find the container with id 48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8 Oct 30 13:03:38.982051 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.982025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:38.990076 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:38.989963 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hjjpd"] Oct 30 13:03:38.993213 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:38.993186 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cd14b5_9230_48f3_8f92_67425d7a0122.slice/crio-35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61 WatchSource:0}: Error finding container 35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61: Status 404 returned error can't find the container with id 35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61 Oct 30 13:03:39.120778 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:39.120746 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:39.174215 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:39.174147 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfvdc"] Oct 30 13:03:39.177401 ip-10-0-130-43 kubenswrapper[2583]: W1030 13:03:39.177373 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c9725b_8217_433e_8d03_d8a493c9f336.slice/crio-700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c WatchSource:0}: Error finding container 700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c: Status 404 returned error can't find the container with id 700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c Oct 30 13:03:39.181521 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:39.181498 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjjpd" event={"ID":"d3cd14b5-9230-48f3-8f92-67425d7a0122","Type":"ContainerStarted","Data":"35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61"} Oct 30 13:03:39.182583 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:39.182560 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kn246" event={"ID":"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1","Type":"ContainerStarted","Data":"48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8"} Oct 30 13:03:40.190282 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:40.190242 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfvdc" event={"ID":"89c9725b-8217-433e-8d03-d8a493c9f336","Type":"ContainerStarted","Data":"700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c"} Oct 30 13:03:40.203362 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:40.203317 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.348798 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:40.348769 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.550960 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:40.550888 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.834417 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:40.834329 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.968762 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:40.968724 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.196503 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:41.196421 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjjpd" event={"ID":"d3cd14b5-9230-48f3-8f92-67425d7a0122","Type":"ContainerStarted","Data":"6e55d94cab1e2bd81ccc02f34fac829677f364ed4ea1e90f599c00e50509dbf5"} Oct 30 13:03:41.196950 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:41.196509 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjjpd" event={"ID":"d3cd14b5-9230-48f3-8f92-67425d7a0122","Type":"ContainerStarted","Data":"af13f7f380a41eb778f8e7d0f6cd59b81d7cb1ac84b1947b17a4533694480e80"} Oct 30 13:03:41.235179 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:41.235128 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hjjpd" podStartSLOduration=67.995550231 podStartE2EDuration="1m9.23511218s" totalImagesPullingTime="1.239561949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:41.23278799 +0000 UTC m=+69.050438983" watchObservedRunningTime="2026-10-30 13:03:41.23511218 +0000 UTC m=+69.052763171" Oct 30 13:03:41.403674 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:41.403529 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.533192 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:41.533149 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:42.205457 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:42.205368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:42.340139 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:42.340101 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:43.096736 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:43.096701 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:43.207034 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:43.206998 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfvdc" event={"ID":"89c9725b-8217-433e-8d03-d8a493c9f336","Type":"ContainerStarted","Data":"313e42087456bf5538a40841cf0b5f2c44392ca9fa2cae0c7d1b6309a112286a"} Oct 30 13:03:43.207717 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:43.207694 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:43.241032 ip-10-0-130-43 kubenswrapper[2583]: I1030 13:03:43.240974 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gfvdc" podStartSLOduration=67.856534614 podStartE2EDuration="1m11.24095545s" totalImagesPullingTime="3.384420836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:43.24092859 +0000 UTC m=+71.058579585" watchObservedRunningTime="2026-10-30 13:03:43.24095545 +0000 UTC m=+71.058606444" Oct 30 13:03:44.067352 ip-10-0-130-43 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 13:03:44.090973 ip-10-0-130-43 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 13:03:44.091143 ip-10-0-130-43 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 13:03:44.091197 ip-10-0-130-43 systemd[1]: kubelet.service: Consumed 4.400s CPU time. Oct 30 13:03:44.102416 ip-10-0-130-43 systemd[1]: Starting Kubernetes Kubelet... Oct 30 13:03:44.164326 ip-10-0-130-43 kubenswrapper[5693]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:03:44.164326 ip-10-0-130-43 kubenswrapper[5693]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 13:03:44.164326 ip-10-0-130-43 kubenswrapper[5693]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:03:44.164326 ip-10-0-130-43 kubenswrapper[5693]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:03:44.166465 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166381 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.166465 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166466 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166470 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166474 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166477 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166480 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166483 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166486 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166489 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166492 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166495 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166497 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166500 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166503 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166506 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166509 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166512 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166515 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166519 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166524 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.166528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166527 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166530 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166533 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166536 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166539 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166542 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166545 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166548 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166550 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166553 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166556 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166558 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166561 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166564 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166566 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166569 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166572 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166576 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166579 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166581 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166584 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.167032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166586 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166590 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166617 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166620 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166623 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166626 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166629 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166631 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166634 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166637 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166639 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166642 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166649 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166651 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166654 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166657 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166659 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166663 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166666 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.167528 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166668 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166671 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166674 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166677 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166679 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166682 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166685 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166687 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166690 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166693 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166696 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166699 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166701 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166705 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166708 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166710 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166713 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166716 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166719 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166722 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.167999 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166725 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166728 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166730 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166733 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166735 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166740 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166743 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166746 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166749 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166752 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166755 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166758 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166760 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166763 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166766 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166769 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166772 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166775 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166777 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.168455 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166780 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166783 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166785 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166788 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166791 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166794 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.166797 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167613 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167622 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167626 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167629 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167632 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167635 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167638 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167641 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167644 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167647 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167650 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167653 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.168917 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167657 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167659 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167662 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167665 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167668 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167670 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167673 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167677 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167681 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167684 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167686 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167689 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167692 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167695 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167698 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167700 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167703 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167705 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167709 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167711 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.169363 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167714 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167717 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167720 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167723 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167725 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167728 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167731 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167737 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167740 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167743 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167746 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167748 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167751 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167754 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167757 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167760 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167762 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167765 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167768 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.169840 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167771 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167774 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167776 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167779 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167782 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167785 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167787 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167790 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167792 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167795 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167798 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167801 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167804 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167806 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167809 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167811 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167814 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167817 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167819 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167823 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.170278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167826 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167829 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167831 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167834 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167837 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167840 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167842 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167846 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167849 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167851 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167854 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167857 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167859 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167862 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167864 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167867 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167870 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167872 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167875 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167878 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.170774 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167880 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167883 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167887 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167891 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167895 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167898 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167901 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167904 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167907 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167910 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167913 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167916 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167919 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167922 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.167925 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168419 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168425 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168428 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168431 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168436 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.171243 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168439 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168442 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168445 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168449 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168451 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168454 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168457 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168460 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168462 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168466 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168468 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168471 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168474 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168476 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168479 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168482 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168485 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168488 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168491 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168496 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.171779 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168499 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168502 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168504 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168507 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168510 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168512 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168515 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168518 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168520 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168523 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168526 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168528 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168531 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168534 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168537 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168540 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168543 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168545 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168548 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.172245 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168551 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168553 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168556 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168558 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168562 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168565 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168567 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168570 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168573 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168575 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168578 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168581 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168585 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168588 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168591 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168606 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168609 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168612 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168615 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168618 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.172704 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168620 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168623 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168626 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168628 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168631 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168634 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168637 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168640 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168643 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168646 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168648 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168651 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168653 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168656 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168659 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168661 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168665 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168669 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168672 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168675 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.173499 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168678 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168681 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168683 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168686 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168690 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168693 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168696 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168699 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168701 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168706 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168709 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168711 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168714 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168717 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168720 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168722 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168725 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168729 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168731 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.174143 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168735 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168737 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.168740 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168840 5693 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168856 5693 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168866 5693 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168872 5693 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168877 5693 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168880 5693 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168885 5693 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168894 5693 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168897 5693 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168901 5693 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168904 5693 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168908 5693 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168911 5693 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168914 5693 flags.go:64] FLAG: --cgroup-root="" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168919 5693 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168922 5693 flags.go:64] FLAG: --client-ca-file="" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168926 5693 flags.go:64] FLAG: --cloud-provider="external" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168929 5693 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168933 5693 flags.go:64] FLAG: --cluster-domain="" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168938 5693 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168941 5693 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 13:03:44.174608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168944 5693 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168949 5693 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168953 5693 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168956 5693 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168960 5693 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168963 5693 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168967 5693 flags.go:64] FLAG: --contention-profiling="false" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168970 5693 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168973 5693 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168977 5693 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168980 5693 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168985 5693 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168989 5693 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168992 5693 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168995 5693 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.168998 5693 flags.go:64] FLAG: --enable-server="true" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169001 5693 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169005 5693 flags.go:64] FLAG: --event-burst="100" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169008 5693 flags.go:64] FLAG: --event-qps="50" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169012 5693 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169015 5693 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169018 5693 flags.go:64] FLAG: --eviction-hard="" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169023 5693 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169026 5693 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169029 5693 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 13:03:44.175166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169035 5693 flags.go:64] FLAG: --eviction-soft="" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169038 5693 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169041 5693 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169045 5693 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169048 5693 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169052 5693 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169056 5693 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169059 5693 flags.go:64] FLAG: --feature-gates="" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169064 5693 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169068 5693 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169071 5693 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169075 5693 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169078 5693 flags.go:64] FLAG: --healthz-port="10248" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169082 5693 flags.go:64] FLAG: --help="false" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169085 5693 flags.go:64] FLAG: --hostname-override="ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169088 5693 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169091 5693 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169095 5693 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169099 5693 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169102 5693 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169106 5693 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169108 5693 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169112 5693 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 13:03:44.175781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169115 5693 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169118 5693 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169121 5693 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169124 5693 flags.go:64] FLAG: --kube-reserved="" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169128 5693 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169131 5693 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169134 5693 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169137 5693 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169140 5693 flags.go:64] FLAG: --lock-file="" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169144 5693 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169148 5693 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169151 5693 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169157 5693 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169161 5693 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169165 5693 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169169 5693 flags.go:64] FLAG: --logging-format="text" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169172 5693 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169176 5693 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169179 5693 flags.go:64] FLAG: --manifest-url="" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169182 5693 flags.go:64] FLAG: --manifest-url-header="" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169187 5693 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169191 5693 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169195 5693 flags.go:64] FLAG: --max-pods="110" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169198 5693 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169202 5693 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 13:03:44.176324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169205 5693 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169209 5693 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169212 5693 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169216 5693 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169219 5693 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169227 5693 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169230 5693 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169234 5693 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169237 5693 flags.go:64] FLAG: --pod-cidr="" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169241 5693 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169244 5693 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169247 5693 flags.go:64] FLAG: --pods-per-core="0" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169250 5693 flags.go:64] FLAG: --port="10250" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169253 5693 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169256 5693 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05eed3293c1f792a6" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169260 5693 flags.go:64] FLAG: --qos-reserved="" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169265 5693 flags.go:64] FLAG: --read-only-port="10255" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169269 5693 flags.go:64] FLAG: --register-node="true" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169272 5693 flags.go:64] FLAG: --register-with-taints="" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169276 5693 flags.go:64] FLAG: --registry-burst="10" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169279 5693 flags.go:64] FLAG: --registry-qps="5" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169283 5693 flags.go:64] FLAG: --reserved-cpus="" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169286 5693 flags.go:64] FLAG: --reserved-memory="" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169290 5693 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169294 5693 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169297 5693 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 13:03:44.176944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169300 5693 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169303 5693 flags.go:64] FLAG: --runonce="false" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169306 5693 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169310 5693 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169313 5693 flags.go:64] FLAG: --seccomp-default="false" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169316 5693 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169319 5693 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169323 5693 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169326 5693 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169329 5693 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169333 5693 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169336 5693 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169339 5693 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169342 5693 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169345 5693 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169349 5693 flags.go:64] FLAG: --system-cgroups="" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169352 5693 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169357 5693 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169360 5693 flags.go:64] FLAG: --tls-cert-file="" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169363 5693 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169367 5693 flags.go:64] FLAG: --tls-min-version="" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169371 5693 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169376 5693 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169379 5693 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169382 5693 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169385 5693 flags.go:64] FLAG: --v="2" Oct 30 13:03:44.177559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169390 5693 flags.go:64] FLAG: --version="false" Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169396 5693 flags.go:64] FLAG: --vmodule="" Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169401 5693 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169404 5693 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169536 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169540 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169543 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169546 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169549 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169552 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169555 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169558 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169560 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169564 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169568 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169571 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169575 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169578 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169581 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169583 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.178161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169586 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169589 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169591 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169606 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169609 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169612 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169615 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169618 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169622 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169625 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169627 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169630 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169633 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169637 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169639 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169642 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169645 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169648 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169651 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169654 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.178648 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169656 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169659 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169662 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169665 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169667 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169670 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169673 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169676 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169679 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169682 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169685 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169688 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169691 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169693 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169696 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169699 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169701 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169704 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169706 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169709 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.179118 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169713 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169716 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169719 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169721 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169724 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169728 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169731 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169734 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169737 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169740 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169742 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169745 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169748 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169750 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169754 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169756 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169759 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169762 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169764 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169767 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.179585 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169770 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169772 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169775 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169778 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169781 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169784 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169787 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169790 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169793 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169797 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169801 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169804 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169809 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169812 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169814 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169817 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169820 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169824 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169827 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.180057 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169830 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169833 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169835 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169839 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169842 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169845 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169848 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169851 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169854 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169856 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169859 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.169886 5693 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169921 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169925 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169929 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169932 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.180517 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169935 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169939 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169942 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169945 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169948 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169950 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169953 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169956 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169958 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169961 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169967 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169970 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169973 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169975 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169980 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169983 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169985 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169988 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169991 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169993 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.180932 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169996 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.169999 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170001 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170004 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170007 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170009 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170012 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170016 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170021 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170024 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170027 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170030 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170032 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170036 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170039 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170041 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170044 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170047 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170049 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.181437 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170052 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170055 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170058 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170062 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170065 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170068 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170070 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170074 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170077 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170080 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170083 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170085 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170088 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170091 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170094 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170096 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170099 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170102 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170106 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.182037 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170110 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170113 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170116 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170119 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170122 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170125 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170128 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170130 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170134 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170137 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170139 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170142 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170145 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170148 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170150 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170153 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170157 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170160 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170162 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170165 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.182586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170169 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170172 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170175 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170177 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170180 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170183 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170186 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170188 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170191 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170194 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170196 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170199 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170202 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170205 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170207 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170210 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170213 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170216 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170220 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170222 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.183134 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170225 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170228 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170231 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.170234 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.175362 5693 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.175380 5693 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.175401 5693 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.175406 5693 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175475 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175480 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175484 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175487 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175490 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175493 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175496 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175499 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175502 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175505 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175508 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175510 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.183692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175513 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175516 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175519 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175527 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175530 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175533 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175536 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175539 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175542 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175546 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175549 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175552 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175555 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175558 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175561 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175564 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175566 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175569 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175573 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.184201 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175578 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175582 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175585 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175588 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175590 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175593 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175611 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175615 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175618 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175622 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175626 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175629 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175632 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175635 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175638 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175642 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175647 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175650 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175653 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175655 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.184689 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175658 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175661 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175664 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175667 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175670 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175672 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175675 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175678 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175681 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175684 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175687 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175690 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175693 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175696 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175699 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175702 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175705 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175708 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175711 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175713 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.185155 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175716 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175719 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175722 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175724 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175727 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175730 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175733 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175735 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175739 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175742 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175745 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175747 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175751 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175753 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175757 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175759 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175762 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175765 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175767 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175770 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.185631 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175773 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175776 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175778 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175781 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175784 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175787 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175790 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175792 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175795 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175798 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175801 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175803 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175806 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175809 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175811 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.175839 5693 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:03:44.186162 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175980 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175984 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175987 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175990 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175993 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.175997 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176000 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176003 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176006 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176009 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176011 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176014 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176017 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176020 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176023 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176025 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176028 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176031 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176033 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176036 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.186555 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176040 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176044 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176048 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176052 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176056 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176060 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176065 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176068 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176073 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176077 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176080 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176083 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176086 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176089 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176092 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176094 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176097 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176100 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176103 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176106 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.187045 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176109 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176114 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176118 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176121 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176124 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176127 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176130 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176132 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176135 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176137 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176140 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176143 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176145 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176148 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176151 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176153 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176156 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176159 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.187519 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176163 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176167 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176170 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176173 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176176 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176179 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176182 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176185 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176188 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176190 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176193 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176196 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176199 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176202 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176205 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176208 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176211 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176213 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176216 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.188100 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176219 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176221 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176224 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176227 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176230 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176232 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176235 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176238 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176240 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176243 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176246 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176249 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176252 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176254 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176257 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176260 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176262 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176265 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176268 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176271 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.188665 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176274 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176277 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176279 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176282 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176285 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176288 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176292 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176297 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176301 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.176329 5693 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176370 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176375 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176382 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176388 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176393 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176396 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:03:44.189457 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176399 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176402 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176405 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176408 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176411 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176414 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176416 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176419 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176422 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176425 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176428 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176431 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176433 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176436 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176439 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176441 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176444 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176447 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176450 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176452 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:03:44.189973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176456 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176460 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176465 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176469 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176473 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176476 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176479 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176482 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176484 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176487 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176490 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176493 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176496 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176498 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176502 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176504 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176507 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176510 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176514 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176518 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:03:44.190450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176522 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176525 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176527 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176530 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176534 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176539 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176543 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176547 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176550 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176552 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176555 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176558 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176561 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176564 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176567 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176570 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176574 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176578 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176581 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:03:44.191088 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176584 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176587 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176589 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176592 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176608 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176613 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176617 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176620 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176623 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176626 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176629 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176631 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176634 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176637 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176640 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176642 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176645 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176648 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176651 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176654 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:03:44.191544 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176656 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176659 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176661 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176669 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176672 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176675 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176678 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176683 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176688 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176692 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176695 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176698 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176701 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176704 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176707 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176710 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176713 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176716 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176719 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176722 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:03:44.192032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:03:44.176725 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.176791 5693 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.177871 5693 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.177932 5693 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.178111 5693 server.go:1014] "Starting client certificate rotation" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.178247 5693 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.178307 5693 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:57:32 +0000 UTC" deadline="2028-07-20 19:48:33.98009302 +0000 UTC" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.178335 5693 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15102h44m49.801759483s" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.178615 5693 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.179718 5693 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.181171 5693 log.go:25] "Validated CRI v1 runtime API" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.183133 5693 log.go:25] "Validated CRI v1 image API" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.183885 5693 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 13:03:44.192501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.186203 5693 fs.go:135] Filesystem UUIDs: map[1d9ba08e-aa38-45eb-b302-67ae7a42efbb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bab503e6-8de1-4b83-90db-b046f47653d4:/dev/nvme0n1p4] Oct 30 13:03:44.192867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.186227 5693 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1262668397b838a46f4210fc7542e2926680afb5f468e17f9b365d8c8ce7558b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1262668397b838a46f4210fc7542e2926680afb5f468e17f9b365d8c8ce7558b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61/userdata/shm major:0 minor:436 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8/userdata/shm major:0 minor:437 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c/userdata/shm major:0 minor:441 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a7cb4e2fb986e6896bcae6c8351e7eb0e9536e18aea9cb565f9bb653865e781/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a7cb4e2fb986e6896bcae6c8351e7eb0e9536e18aea9cb565f9bb653865e781/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd/userdata/shm major:0 minor:424 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cabbb1b2249abb2037929bfff49c21ea7a86ba4aa2d5fee991f280b160b000b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cabbb1b2249abb2037929bfff49c21ea7a86ba4aa2d5fee991f280b160b000b1/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e9b6ca-1d68-43eb-a809-24debe93ae5b/volumes/kubernetes.io~projected/kube-api-access-qblw7:{mountpoint:/var/lib/kubelet/pods/05e9b6ca-1d68-43eb-a809-24debe93ae5b/volumes/kubernetes.io~projected/kube-api-access-qblw7 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e9b6ca-1d68-43eb-a809-24debe93ae5b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/05e9b6ca-1d68-43eb-a809-24debe93ae5b/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc/volumes/kubernetes.io~projected/kube-api-access-g884v:{mountpoint:/var/lib/kubelet/pods/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc/volumes/kubernetes.io~projected/kube-api-access-g884v major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/134089b3-8c7a-4f4b-8330-127fb60ba056/volumes/kubernetes.io~projected/kube-api-access-qbjlz:{mountpoint:/var/lib/kubelet/pods/134089b3-8c7a-4f4b-8330-127fb60ba056/volumes/kubernetes.io~projected/kube-api-access-qbjlz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~projected/kube-api-access-npntf:{mountpoint:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~projected/kube-api-access-npntf major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19c122c0-62b5-437d-91d5-3d519c7958f7/volumes/kubernetes.io~projected/kube-api-access-bp6bs:{mountpoint:/var/lib/kubelet/pods/19c122c0-62b5-437d-91d5-3d519c7958f7/volumes/kubernetes.io~projected/kube-api-access-bp6bs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~projected/kube-api-access-27z9n:{mountpoint:/var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~projected/kube-api-access-27z9n major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34b15028-7cbf-46f3-9ffd-d9a8369128af/volumes/kubernetes.io~projected/kube-api-access-zfbkp:{mountpoint:/var/lib/kubelet/pods/34b15028-7cbf-46f3-9ffd-d9a8369128af/volumes/kubernetes.io~projected/kube-api-access-zfbkp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48592583-022f-490d-b547-a4876a43da65/volumes/kubernetes.io~projected/kube-api-access-dcsqn:{mountpoint:/var/lib/kubelet/pods/48592583-022f-490d-b547-a4876a43da65/volumes/kubernetes.io~projected/kube-api-access-dcsqn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5276b8dc-f3cf-4d3b-a630-bd16aca9a373/volumes/kubernetes.io~projected/kube-api-access-pw4f4:{mountpoint:/var/lib/kubelet/pods/5276b8dc-f3cf-4d3b-a630-bd16aca9a373/volumes/kubernetes.io~projected/kube-api-access-pw4f4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~projected/kube-api-access-mm2pl:{mountpoint:/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~projected/kube-api-access-mm2pl major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~projected/kube-api-access-v2jtv:{mountpoint:/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~projected/kube-api-access-v2jtv major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~projected/kube-api-access-hwm5n:{mountpoint:/var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~projected/kube-api-access-hwm5n major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~projected/kube-api-access-pgpr7:{mountpoint:/var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~projected/kube-api-access-pgpr7 major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89c9725b-8217-433e-8d03-d8a493c9f336/volumes/kubernetes.io~projected/kube-api-access-mbv2m:{mountpoint:/var/lib/kubelet/pods/89c9725b-8217-433e-8d03-d8a493c9f336/volumes/kubernetes.io~projected/kube-api-access-mbv2m major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~projected/kube-api-access-6wfqr:{mountpoint:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~projected/kube-api-access-6wfqr major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~projected/kube-api-access-7jbcd:{mountpoint:/var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~projected/kube-api-access-7jbcd major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3cd14b5-9230-48f3-8f92-67425d7a0122/volumes/kubernetes.io~projected/kube-api-access-cm7db:{mountpoint:/var/lib/kubelet/pods/d3cd14b5-9230-48f3-8f92-67425d7a0122/volumes/kubernetes.io~projected/kube-api-access-cm7db major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3cd14b5-9230-48f3-8f92-67425d7a0122/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d3cd14b5-9230-48f3-8f92-67425d7a0122/volumes/kubernetes.io~secret/metrics-certs major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da949e5b-eb60-4823-b417-41067e8d0959/volumes/kubernetes.io~projected/kube-api-access-5tcpr:{mountpoint:/var/lib/kubelet/pods/da949e5b-eb60-4823-b417-41067e8d0959/volumes/kubernetes.io~projected/kube-api-access-5tcpr major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da949e5b-eb60-4823-b417-41067e8d0959/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/da949e5b-eb60-4823-b417-41067e8d0959/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~projected/kube-api-access-g84ql:{mountpoint:/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~projected/kube-api-access-g84ql major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~projected/kube-api-access-j6nzl:{mountpoint:/var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~projected/kube-api-access-j6nzl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e85daeb3-1320-4a20-b831-0817f651f8e8/volumes/kubernetes.io~projected/kube-api-access-hbkc7:{mountpoint:/var/lib/kubelet/pods/e85daeb3-1320-4a20-b831-0817f651f8e8/volumes/kubernetes.io~projected/kube-api-access-hbkc7 major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e85daeb3-1320-4a20-b831-0817f651f8e8/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/e85daeb3-1320-4a20-b831-0817f651f8e8/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volumes/kubernetes.io~projected/kube-api-access-nm8zj:{mountpoint:/var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volumes/kubernetes.io~projected/kube-api-access-nm8zj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fec11257-4c9d-48b9-9333-970360865ea7/volumes/kubernetes.io~projected/kube-api-access-dh8jb:{mountpoint:/var/lib/kubelet/pods/fec11257-4c9d-48b9-9333-970360865ea7/volumes/kubernetes.io~projected/kube-api-access-dh8jb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~projected/kube-api-access-2pt8t:{mountpoint:/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~projected/kube-api-access-2pt8t major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:236 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/55391332ddb732ec6f5f95d08cce14249f4204d7edf0d30aa6b555f728f0b995/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ea069c8eea62c9bd685f3528965d5f04aac7757b34d86e86e5b68dd46c38ac25/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/33de279d292eef1768b9b9c0f0f4569c3fb4f0adb1e505297fba163bb697e72a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3f1dc2c999296346b797a57fe10f6036642a09121eeda445c334212bee9bd362/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/eb863856f72fc3240a9e0af0ce4a791f890138337be5bcf4a459719b3eab7bec/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/26bf670681e8f267940e7af9cc7bfb230602ab75e6d1b870b1146539e294d39d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f86b5a600adb68228ca458f9b2de1027683d1ae81375c559456cb36716241c26/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/923ae9eb9d90684fb6fbfbc86f7214bbe6ac6884c0839ddfe2cbe75b2b79bbd4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/024879f8f83ce2cfd743291445b76adcac18931bb7599513cc207014459876ab/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/31e34be47085009eca897270026444e23fc33ecd5b10483be0c3c05e2bd59403/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2fe2bb6f270cd7ca4239a2ae6e25cb10063addb84bc5c18a90a7b7d981511b7d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/b63eadeba5352a7c0c95c5a7e76dedf068d6cf7db1e7ebbe043c2224878d710e/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c2e9c0bc247e4a904920fc6562ae94bfb5b71aa717fdcf2bf799e55a87f21bd6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6b9c76b2ac87ced0d32c596394c0879134712a5401d5c3721d4ab58717905ec5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/90faef201028659bbe6e427e76335d49cb95e8a0af0873d007d467acddba7871/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4cb5e118b7fc615f64b18447a147e8391840a21737b5dd92917def61cb121a29/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ab889499d19d17c4fdb79c0833f53ffdaa202f7c4dc0bbdbda206ee21b54410d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8629f2b78be5ec8dfa4391c6a094159a6e4ee5361bd129ba856e7d4ad41de25f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bb503f4b5b57580aea0335233c084747370f67f60e1b4723c02701fea9daa3fc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9ae60e8d2ebdf54a74423e773acf15a1aeeabbb04adeff658d93d564ebaae980/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/10a51e79bba80db0302b2e2ab9154d4e0e193bc414ecdb9af4051586bd9518f7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/a3a95c80e13ce7501293691a0df3cb5920023b20876a22e5375f05ecc1f621fe/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/76fa6054d292f8c28ce7ff2659b385a90033dc74924125b4145e308b5cf65178/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/7b461e3f3aeafc55bb0733ff3129ec7a5946ab103574b8c16f02028c51cf085e/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/6f8544a997621363b61ef541b93e1d6bb8025d5f792706725060b26c4bc42f5f/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/22765ee45d724f0adfd20e901bd3e8915d4269baa10ed0eb2b995af5940e229c/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/ea08583aca022da91aa40f65f469dd3c06b46f2bb6701748057ef86edc870678/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/f6458665937e3c7ec64b080dac9709129b47e16fc5b6bc0bb96381e4315c4d90/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/788046dd90edbab6cc63fcedae0d464c744a6d2b347490bcffe9dfa0e612b25b/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b699692eb5669fe8e65e09428c85bff1a09b476532e97d0c80711fef777dad25/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/28bf0c1c6908f44f774180bd113e809d056a4339622331726058f20dbb5c49b0/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/fe9e6662ca4772620fba8e1df64e077ededfab566c6a61cb4f884b35dafd4f9d/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/e7332113bdbc1a735dfe3d97abeab37f71f943892e818edb6b3e230c9572613d/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/0d64f6c14918c0240981588794a7dcbaba013520646277bde6a131f116e9d6ad/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/71fc8f2f3fec82eaabc3daae04cf8f9e799832b3604101c22834d918304d3ebc/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/3045f321cb1f05d442acb20eeaad684cd38ad53abf3923e161fbbeb66d3c6c2e/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/efee4e8c451a2bbdcb202f774e63ef027a80374aa2b502b6a644cf2961a6eb40/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/7c4fefbeccdc8b22693e3957ee09d869aec004e3cb749f28e9f1be3eeea04719/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/e9dd79ca49998727836a772d3ebf7187ba158933477a69232e758e2d1c0f0db1/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/8e71648e4541a12880197cb780ab1ea8b0a01b17170a9b5afd768bcf9115826c/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/342ef245e552ee5b4125235d3c4fb2898ea4e3851024f50a54959b53f6253fd0/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/997cccba1a21cc9a0e8390e3db5dfd3e58b61764ae972d09346a49f2a014dd7c/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/8dfd7b9c4a95f758320260a8d2888bde1c76a97095e6a8174ed8c08137c7dc79/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/ce1de5077ad423f28f9b31b6d216345aea5db08e35605fb8bb55f84394105db4/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/0aea83f37f8f0186bf95335b5beec0aae9902b19707df5bb578676d8d6effe36/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/e262c60e4b8451725cc256b62a75ffaba5dac4b9d5512d287cb280d25d3c0f92/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/6a5670e9b4c09504bd44b815be061de22b2afccbe9f0151a124ee50639d7fc37/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/eed02cc92a0cc1b1c5a877b218ba78498df44f942af6f806e71430852a40410b/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/df40c010de6ec37f3655deb44cea1bf60ba3c4d90dcd87792a29fb841b2b8d40/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/c0c996e1d1d35d4a559219ae8247dde73e2a54fcd07e6845c4b9f07f21a71ffb/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/ed8c3b23cbbc9a12025061e30d4e842196bb0d3fc95ca5e41179cebde4cebacc/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/fc38083c05725c6e296636cf3dc8bde7fc411b9901ec606000ff10acaff30576/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/82e56a5176b70677ab3fb9a59ebaeaf094d57b782af49d945cb54ae795963b41/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/f176dff86e9b28760d479b9b1f84b960a607ca64dbc581c90b3163971b6bb1d5/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/914eb4d6d75dba042ae41fed56837ec7c7b679f9ff1b0fc03c2c1e6647d2d4d6/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/a2602fab3a84b5ab70382c44ead0ef759f31af28c428713e7e817b36bb8194da/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/17f7650e679a5061efe2fa597688076fc6231afd3ad07989dc034e3ea254a2f7/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/d463f672ed91de36aa25d62949666b01cbbab7f0bb30416a60b69f03b5566435/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/6eeb797148c54cccb11d02f5fbbe50167961b2a53e2895217d660440d2837dd6/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/4b8354d512995f4666f8063996b36c558cefd4a22cf17fc5c249f93a7cb1fb46/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/ef97d56a311d26e30430c4f654413ddbadf358da61e6dd6fbc903783b603b4be/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/2fd569dc96a4f0b7ff5df4b8e670fde3ba9af0578cb0177574ff85444539e958/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/4fb130deb4224e0c4224663e3b51498ddd0733f353388ce6d7ac0e47421a3a67/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/87bb945f5d70c1dcbf976daa8289e4bf7ae5e7f46b2c626d37553f8bbac0b5ef/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/d214413ceb39553ab5bc60531ed830d3c3f2678607637e659ac4be12599a6bb0/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/97cc106a4b3751c03f095790308e111076505dfe0943959e3f5cd6baab6fdd45/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/b8157927d37bbda1aa1c483e9ce59468b3cc677a1c7e3aa3f589941ac44e3abf/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/fa3fa6e45de5f9b4f006beda05dec41ecfc76938566de6a631a6287ffde1492b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a31e26dad5702566d502b0060fa7010627265e09c35530e297a25050d786dfbd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/18ff4f2056510a5a96d0ab456d16d3596a866b544dd03b799e5beaa4ac76256b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/a5d52eebb286972eacc2b8b3a3d5a3cec80ecdcca5a21b39c881c36748c661ef/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3992637ad5770722b163c3f423f45a60c713810d2450322a8e6a1b6c6003a68b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/35dea284d529025159b6a17198c5aa96e28376160c4ef9ff94927ec2d7f944c0/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/d34f8604e87de584463d11f211fcdf9857c3979d73dde6436c8995e71a214c9c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/1c001d3130ef96066ecff7b39a84f474b7b8cbde8e70bf98f40845e94095cc05/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/01780110d43c149c48e8f91eda60f4112b6934e6a8b811cacca501a9a2840ce3/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/9611650eccbcc399b6340ddb2fa4b1264d7972e5b91588dee1ae7a0b82dbf025/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 13:03:44.196363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.195949 5693 manager.go:217] Machine: {Timestamp:2026-10-30 13:03:44.195448824 +0000 UTC m=+0.062067007 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134503 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f31a88060c1433c340b413b36d489 SystemUUID:ec2f31a8-8060-c143-3c34-0b413b36d489 BootID:91e78db8-c768-4882-ae2c-468bfc3bbbe0 Filesystems:[{Device:/run/containers/storage/overlay-containers/7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f6b7bf12536b7837ac1ac5f31da8afae8ed23036a39000fd5ebbff7207eac43/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~projected/kube-api-access-j6nzl DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:435 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~projected/kube-api-access-pgpr7 DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~projected/kube-api-access-g84ql DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a7cb4e2fb986e6896bcae6c8351e7eb0e9536e18aea9cb565f9bb653865e781/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~projected/kube-api-access-mm2pl DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~projected/kube-api-access-27z9n DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc/volumes/kubernetes.io~projected/kube-api-access-g884v DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d06693dd980ce99795c4c86e4f10471709f56dca116e9c641fb228c5a33fe92/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e31119d742d64dc67b6415e237dcbe7a01d56cd34ee9043b2acfd3a708c386dd/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd/userdata/shm DeviceMajor:0 DeviceMinor:424 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23c0f661581a7188143ba7478d3c83585263fa93268b7a83f48450e55b7dd350/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a9149e3f3870d5f6c29d9391d0c54bb21eddd1fb1c4ccd42f94bc4dc30c9b41/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~projected/kube-api-access-6wfqr DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~projected/kube-api-access-7jbcd DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00e981e6f04926bf80331cc3758566515e68336573cbe3303d6acdab428a347a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca8a3b3-0f1e-4bd2-8660-6d8add527517/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volumes/kubernetes.io~projected/kube-api-access-nm8zj DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eca677810c9089fbd1957af95cf14769a4d4676d3697cb0502cce6b729c479d/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:381 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35bd426d37026a96f83059db3164d4068435c1d45d725abae25131e9d9cb1e61/userdata/shm DeviceMajor:0 DeviceMinor:436 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~projected/kube-api-access-npntf DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5b6492e56e7e793d395cc8f7215c69e0ee837c7778d141b1e331622d0f9749a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/da949e5b-eb60-4823-b417-41067e8d0959/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes/kubernetes.io~projected/kube-api-access-2pt8t DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fec11257-4c9d-48b9-9333-970360865ea7/volumes/kubernetes.io~projected/kube-api-access-dh8jb DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34b15028-7cbf-46f3-9ffd-d9a8369128af/volumes/kubernetes.io~projected/kube-api-access-zfbkp DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7154179d3db570f61d3e719c63f607db1c489c704ca93e145fd21e51c18aa3c/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~projected/kube-api-access-hwm5n DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bdfaae46-aa7b-4311-8ea5-1f6716311a36/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/187fc60f-7b2a-45dd-b5ff-13839099e07f/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e85daeb3-1320-4a20-b831-0817f651f8e8/volumes/kubernetes.io~projected/kube-api-access-hbkc7 DeviceMajor:0 DeviceMinor:364 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5433b985294a87002f57cac23c8607716da7b63a151a8bda9f4794dad3cfcb3b/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da949e5b-eb60-4823-b417-41067e8d0959/volumes/kubernetes.io~projected/kube-api-access-5tcpr DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cabbb1b2249abb2037929bfff49c21ea7a86ba4aa2d5fee991f280b160b000b1/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/700b4e43f17442b352bb5f4d8cc2e727ad7703725ecd8377a004884eec4aff8c/userdata/shm DeviceMajor:0 DeviceMinor:441 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/342d2b75878caa564f8aba300ceff75047dca0362d4613dbd7de24d6b96437f2/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/134089b3-8c7a-4f4b-8330-127fb60ba056/volumes/kubernetes.io~projected/kube-api-access-qbjlz DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19c122c0-62b5-437d-91d5-3d519c7958f7/volumes/kubernetes.io~projected/kube-api-access-bp6bs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3cd14b5-9230-48f3-8f92-67425d7a0122/volumes/kubernetes.io~projected/kube-api-access-cm7db DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05e9b6ca-1d68-43eb-a809-24debe93ae5b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5276b8dc-f3cf-4d3b-a630-bd16aca9a373/volumes/kubernetes.io~projected/kube-api-access-pw4f4 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05e9b6ca-1d68-43eb-a809-24debe93ae5b/volumes/kubernetes.io~projected/kube-api-access-qblw7 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3cd14b5-9230-48f3-8f92-67425d7a0122/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:430 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4012f5224050177030a4f06bad4874af86531d04871a07176dad7ea0ce715f9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbc1dd68-9c18-40cf-9d70-c812d51564fc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78a085e4-85e9-466c-984c-e08ccc0b8862/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e23963256863ab777fea14265b247a16c4317e437b66aec0a0ebe5a0b8f91a3/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e86df5d-83ed-433e-9c0e-b5f59970dfe9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:422 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89c9725b-8217-433e-8d03-d8a493c9f336/volumes/kubernetes.io~projected/kube-api-access-mbv2m DeviceMajor:0 DeviceMinor:440 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48592583-022f-490d-b547-a4876a43da65/volumes/kubernetes.io~projected/kube-api-access-dcsqn DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d47b29af4184564eb08ab33fd4382989582bd7b296123bad9654b6bb272ca04b/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e85daeb3-1320-4a20-b831-0817f651f8e8/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1262668397b838a46f4210fc7542e2926680afb5f468e17f9b365d8c8ce7558b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:418 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a56d5807-0ea6-48b5-8487-64330925498a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25d7c378f9f74a58ba178554c62592e5128e37a4b54c9ab9fd628b1fe46cee21/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes/kubernetes.io~projected/kube-api-access-v2jtv DeviceMajor:0 DeviceMinor:382 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48b2b8608fa475954b89a1dea587056e98baeb0270ef63062a1766677fc82fd8/userdata/shm DeviceMajor:0 DeviceMinor:437 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e060d933-f69d-44fd-a416-6a9877c0cfed/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00e981e6f04926b MacAddress:86:90:13:56:b3:bf Speed:10000 Mtu:8901} {Name:25d7c378f9f74a5 MacAddress:62:3a:43:d0:df:37 Speed:10000 Mtu:8901} {Name:342d2b75878caa5 MacAddress:66:59:01:6b:46:60 Speed:10000 Mtu:8901} {Name:35bd426d37026a9 MacAddress:22:a1:ee:7a:3b:00 Speed:10000 Mtu:8901} {Name:48b2b8608fa4759 MacAddress:fe:fa:75:18:4d:35 Speed:10000 Mtu:8901} {Name:4a9149e3f3870d5 MacAddress:26:34:27:00:b3:11 Speed:10000 Mtu:8901} {Name:5c1108eb9fb8022 MacAddress:3a:8e:53:6a:9e:47 Speed:10000 Mtu:8901} {Name:6eca677810c9089 MacAddress:ee:8a:9a:5e:6b:9b Speed:10000 Mtu:8901} {Name:700b4e43f17442b MacAddress:1a:30:2a:b9:54:7a Speed:10000 Mtu:8901} {Name:73aaa0154c6f603 MacAddress:4a:0f:bf:90:71:84 Speed:10000 Mtu:8901} {Name:7e23963256863ab MacAddress:5a:f3:e7:7d:fc:37 Speed:10000 Mtu:8901} {Name:a63fe62f46baa05 MacAddress:e2:3a:af:0c:55:35 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c4:f5:d2:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:68:c2:8c:e0:34 Speed:0 Mtu:8901} {Name:cabbb1b2249abb2 MacAddress:ce:54:b1:99:9e:78 Speed:10000 Mtu:8901} {Name:cbbe48d37341e7d MacAddress:22:4a:0a:78:1c:54 Speed:10000 Mtu:8901} {Name:d47b29af4184564 MacAddress:aa:8e:06:3d:e9:89 Speed:10000 Mtu:8901} {Name:e31119d742d64dc MacAddress:82:41:b2:ab:de:8a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c4:f5:d2:a3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:fe:9f:a2:19:68:aa Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:57:18:db:1d:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 13:03:44.196593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196367 5693 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 13:03:44.196593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196412 5693 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 13:03:44.196786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196758 5693 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 13:03:44.196912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196786 5693 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 13:03:44.196956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196915 5693 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 13:03:44.196956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196922 5693 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 13:03:44.196956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196932 5693 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:03:44.196956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.196945 5693 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:03:44.197053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197030 5693 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 13:03:44.197218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197209 5693 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 13:03:44.197321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197312 5693 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 13:03:44.197355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197347 5693 kubelet.go:493] "Attempting to sync node with API server" Oct 30 13:03:44.197382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197360 5693 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 13:03:44.197382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197371 5693 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 13:03:44.197382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197380 5693 kubelet.go:405] "Adding apiserver pod source" Oct 30 13:03:44.197457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.197388 5693 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 13:03:44.198336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.198318 5693 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 13:03:44.198336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.198336 5693 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:03:44.198905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.198885 5693 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 13:03:44.201548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.201528 5693 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 13:03:44.201548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.201544 5693 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:03:44.201707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.201672 5693 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 13:03:44.201900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.201885 5693 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 13:03:44.201980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.201925 5693 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 13:03:44.202054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202042 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 13:03:44.202108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202062 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 13:03:44.202108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202073 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 13:03:44.202108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202083 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 13:03:44.202108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202093 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 13:03:44.202108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202103 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202113 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202123 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202134 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202144 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202157 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202172 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202207 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 13:03:44.202374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.202218 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 13:03:44.205676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.205660 5693 server.go:1263] "Started kubelet" Oct 30 13:03:44.205763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.205731 5693 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 13:03:44.206229 ip-10-0-130-43 systemd[1]: Started Kubernetes Kubelet. Oct 30 13:03:44.206877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.206691 5693 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 13:03:44.206877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.206833 5693 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 13:03:44.210623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.208015 5693 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 13:03:44.210623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.209208 5693 server.go:317] "Adding debug handlers to kubelet server" Oct 30 13:03:44.212225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212206 5693 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 13:03:44.212314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212230 5693 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 13:03:44.212314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212292 5693 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 13:03:44.212314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212308 5693 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 13:03:44.212484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212292 5693 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:34 +0000 UTC" deadline="2028-05-17 19:44:15.114322155 +0000 UTC" Oct 30 13:03:44.212484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212327 5693 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13566h40m30.901998104s" Oct 30 13:03:44.212484 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.212372 5693 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-43.ec2.internal\" not found" Oct 30 13:03:44.212484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.212408 5693 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215179 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215211 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e060d933-f69d-44fd-a416-6a9877c0cfed" volumeName="kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-tuned" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215225 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbc1dd68-9c18-40cf-9d70-c812d51564fc" volumeName="kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-config" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215233 5693 factory.go:55] Registering systemd factory Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215261 5693 factory.go:223] Registration of the systemd container factory successfully Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215238 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbc1dd68-9c18-40cf-9d70-c812d51564fc" volumeName="kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-env-overrides" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215324 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbc1dd68-9c18-40cf-9d70-c812d51564fc" volumeName="kubernetes.io/projected/fbc1dd68-9c18-40cf-9d70-c812d51564fc-kube-api-access-nm8zj" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215345 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/empty-dir/187fc60f-7b2a-45dd-b5ff-13839099e07f-audit-log" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215358 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc" volumeName="kubernetes.io/configmap/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215371 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e86df5d-83ed-433e-9c0e-b5f59970dfe9" volumeName="kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215388 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da949e5b-eb60-4823-b417-41067e8d0959" volumeName="kubernetes.io/configmap/da949e5b-eb60-4823-b417-41067e8d0959-config-volume" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215400 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215414 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215434 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215447 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e86df5d-83ed-433e-9c0e-b5f59970dfe9" volumeName="kubernetes.io/projected/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-kube-api-access-hwm5n" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215458 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b15028-7cbf-46f3-9ffd-d9a8369128af" volumeName="kubernetes.io/projected/34b15028-7cbf-46f3-9ffd-d9a8369128af-kube-api-access-zfbkp" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215467 5693 factory.go:153] Registering CRI-O factory Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215472 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="134089b3-8c7a-4f4b-8330-127fb60ba056" volumeName="kubernetes.io/projected/134089b3-8c7a-4f4b-8330-127fb60ba056-kube-api-access-qbjlz" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215482 5693 factory.go:223] Registration of the crio container factory successfully Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215484 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48592583-022f-490d-b547-a4876a43da65" volumeName="kubernetes.io/projected/48592583-022f-490d-b547-a4876a43da65-kube-api-access-dcsqn" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215497 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5276b8dc-f3cf-4d3b-a630-bd16aca9a373" volumeName="kubernetes.io/configmap/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-serviceca" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215524 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215542 5693 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215546 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215558 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdfaae46-aa7b-4311-8ea5-1f6716311a36" volumeName="kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215568 5693 factory.go:103] Registering Raw factory Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215570 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e9b6ca-1d68-43eb-a809-24debe93ae5b" volumeName="kubernetes.io/projected/05e9b6ca-1d68-43eb-a809-24debe93ae5b-kube-api-access-qblw7" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215584 5693 manager.go:1199] Started watching for new ooms in manager Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215582 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c122c0-62b5-437d-91d5-3d519c7958f7" volumeName="kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215619 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c122c0-62b5-437d-91d5-3d519c7958f7" volumeName="kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-binary-copy" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215632 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215646 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e86df5d-83ed-433e-9c0e-b5f59970dfe9" volumeName="kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215662 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a085e4-85e9-466c-984c-e08ccc0b8862" volumeName="kubernetes.io/configmap/78a085e4-85e9-466c-984c-e08ccc0b8862-metrics-client-ca" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215676 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3cd14b5-9230-48f3-8f92-67425d7a0122" volumeName="kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215708 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215737 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aaa6be8-1e4d-44a0-91af-6a98718ba6e8" volumeName="kubernetes.io/configmap/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-konnectivity-ca" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215749 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc" volumeName="kubernetes.io/empty-dir/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-data-volume" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215764 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215777 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c122c0-62b5-437d-91d5-3d519c7958f7" volumeName="kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215798 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" volumeName="kubernetes.io/projected/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-kube-api-access-27z9n" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215815 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215836 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca" seLinuxMountContext="" Oct 30 13:03:44.215827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215852 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215871 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc" volumeName="kubernetes.io/secret/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215887 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" volumeName="kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-metrics-client-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215926 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48592583-022f-490d-b547-a4876a43da65" volumeName="kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-multus-daemon-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215942 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215955 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215967 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215978 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.215990 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216001 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216014 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-metrics-client-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216028 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-federate-client-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216042 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3cd14b5-9230-48f3-8f92-67425d7a0122" volumeName="kubernetes.io/projected/d3cd14b5-9230-48f3-8f92-67425d7a0122-kube-api-access-cm7db" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216054 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbc1dd68-9c18-40cf-9d70-c812d51564fc" volumeName="kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-script-lib" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216069 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbc1dd68-9c18-40cf-9d70-c812d51564fc" volumeName="kubernetes.io/secret/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216086 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fec11257-4c9d-48b9-9333-970360865ea7" volumeName="kubernetes.io/empty-dir/fec11257-4c9d-48b9-9333-970360865ea7-tmp-dir" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216101 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216115 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216136 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e060d933-f69d-44fd-a416-6a9877c0cfed" volumeName="kubernetes.io/projected/e060d933-f69d-44fd-a416-6a9877c0cfed-kube-api-access-j6nzl" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216148 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e85daeb3-1320-4a20-b831-0817f651f8e8" volumeName="kubernetes.io/projected/e85daeb3-1320-4a20-b831-0817f651f8e8-kube-api-access-hbkc7" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216160 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fec11257-4c9d-48b9-9333-970360865ea7" volumeName="kubernetes.io/projected/fec11257-4c9d-48b9-9333-970360865ea7-kube-api-access-dh8jb" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216183 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216195 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" volumeName="kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216209 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216220 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89c9725b-8217-433e-8d03-d8a493c9f336" volumeName="kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216233 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216245 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/projected/a56d5807-0ea6-48b5-8487-64330925498a-kube-api-access-6wfqr" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216256 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216268 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdfaae46-aa7b-4311-8ea5-1f6716311a36" volumeName="kubernetes.io/projected/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-api-access-7jbcd" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216281 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc" volumeName="kubernetes.io/projected/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-api-access-g884v" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216295 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216308 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c122c0-62b5-437d-91d5-3d519c7958f7" volumeName="kubernetes.io/projected/19c122c0-62b5-437d-91d5-3d519c7958f7-kube-api-access-bp6bs" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216319 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" volumeName="kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216333 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e86df5d-83ed-433e-9c0e-b5f59970dfe9" volumeName="kubernetes.io/configmap/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-metrics-client-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216346 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a085e4-85e9-466c-984c-e08ccc0b8862" volumeName="kubernetes.io/projected/78a085e4-85e9-466c-984c-e08ccc0b8862-kube-api-access-pgpr7" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216359 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a085e4-85e9-466c-984c-e08ccc0b8862" volumeName="kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216371 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdfaae46-aa7b-4311-8ea5-1f6716311a36" volumeName="kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216384 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/projected/187fc60f-7b2a-45dd-b5ff-13839099e07f-kube-api-access-npntf" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216396 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e9b6ca-1d68-43eb-a809-24debe93ae5b" volumeName="kubernetes.io/secret/05e9b6ca-1d68-43eb-a809-24debe93ae5b-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216408 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad840e4-950d-4761-a526-65c645e7de03" volumeName="kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216421 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da949e5b-eb60-4823-b417-41067e8d0959" volumeName="kubernetes.io/projected/da949e5b-eb60-4823-b417-41067e8d0959-kube-api-access-5tcpr" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216434 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da949e5b-eb60-4823-b417-41067e8d0959" volumeName="kubernetes.io/secret/da949e5b-eb60-4823-b417-41067e8d0959-metrics-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216446 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216459 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216480 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdfaae46-aa7b-4311-8ea5-1f6716311a36" volumeName="kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216495 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216508 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e85daeb3-1320-4a20-b831-0817f651f8e8" volumeName="kubernetes.io/secret/e85daeb3-1320-4a20-b831-0817f651f8e8-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216523 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216535 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216547 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48592583-022f-490d-b547-a4876a43da65" volumeName="kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-cni-binary-copy" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216560 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216572 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a085e4-85e9-466c-984c-e08ccc0b8862" volumeName="kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216583 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-client-tls" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216033 5693 manager.go:319] Starting recovery of all containers Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216618 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdfaae46-aa7b-4311-8ea5-1f6716311a36" volumeName="kubernetes.io/empty-dir/bdfaae46-aa7b-4311-8ea5-1f6716311a36-volume-directive-shadow" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216640 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216652 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216664 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" volumeName="kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216678 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56d5807-0ea6-48b5-8487-64330925498a" volumeName="kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216701 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216749 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed238e7-3fbe-47d0-8748-eafbed34a837" volumeName="kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216765 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aaa6be8-1e4d-44a0-91af-6a98718ba6e8" volumeName="kubernetes.io/secret/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-agent-certs" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216780 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="134089b3-8c7a-4f4b-8330-127fb60ba056" volumeName="kubernetes.io/configmap/134089b3-8c7a-4f4b-8330-127fb60ba056-iptables-alerter-script" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216803 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="187fc60f-7b2a-45dd-b5ff-13839099e07f" volumeName="kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-client-ca-bundle" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216831 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" volumeName="kubernetes.io/empty-dir/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-textfile" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216843 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" volumeName="kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216859 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5276b8dc-f3cf-4d3b-a630-bd16aca9a373" volumeName="kubernetes.io/projected/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-kube-api-access-pw4f4" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216873 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" volumeName="kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216886 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e060d933-f69d-44fd-a416-6a9877c0cfed" volumeName="kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-tmp" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216898 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdfaae46-aa7b-4311-8ea5-1f6716311a36" volumeName="kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-metrics-client-ca" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216913 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6aaafde-99f7-4cdb-9c95-381d18e5f1b1" volumeName="kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216925 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da949e5b-eb60-4823-b417-41067e8d0959" volumeName="kubernetes.io/empty-dir/da949e5b-eb60-4823-b417-41067e8d0959-tmp-dir" seLinuxMountContext="" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216946 5693 reconstruct.go:97] "Volume reconstruction finished" Oct 30 13:03:44.217934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.216954 5693 reconciler.go:29] "Reconciler: start to sync state" Oct 30 13:03:44.244768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.244712 5693 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 13:03:44.246108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.246083 5693 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 13:03:44.246202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.246115 5693 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 13:03:44.246202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.246138 5693 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 13:03:44.246202 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.246183 5693 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 13:03:44.254111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.254071 5693 generic.go:356] "Generic (PLEG): container finished" podID="146c4e2eccc94fc30a7e9ef0c46e8801" containerID="4f5c6ceff6f29ec7c216d9ef79e526c941363e35e6532491570e786fb20fe62d" exitCode=0 Oct 30 13:03:44.257830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.257699 5693 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 13:03:44.258211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.258175 5693 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:03:44.266946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.266921 5693 generic.go:356] "Generic (PLEG): container finished" podID="2ca8a3b3-0f1e-4bd2-8660-6d8add527517" containerID="efc3fb82ddf97c403f382cb042d90c3f2d483907f48832175da49e1e33fbea41" exitCode=0 Oct 30 13:03:44.269695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.269673 5693 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:03:44.270999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.270973 5693 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="05149c82ff9699008e813c5c89a0ee1d3d5310cdb3cc6cb2a5eb5df1c02bef10" exitCode=0 Oct 30 13:03:44.270999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.270997 5693 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="4651204352d563d906c39192f7b94e500112bcd199bfa997036f652dd6d8c0e3" exitCode=0 Oct 30 13:03:44.271101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.271007 5693 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="a7029f5c88655fce5df73670e7cc0cec52d53cc755daddbef5b9c28b9d6d2e2b" exitCode=0 Oct 30 13:03:44.271101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.271018 5693 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="83261853fa0fcb91dc49575a0af82a843be87bbbed0f96a4619055c4ccce06f3" exitCode=0 Oct 30 13:03:44.271101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.271026 5693 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="53772068dd80f0bc5f9ce428dfc54fb61b382e61c2a6a2b142b7f02911818739" exitCode=0 Oct 30 13:03:44.271101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.271036 5693 generic.go:356] "Generic (PLEG): container finished" podID="19c122c0-62b5-437d-91d5-3d519c7958f7" containerID="85d0bc06002f9ac7622f0cce3198080952cf6c1db3bf38d7ff5e56fd8a2f4b31" exitCode=0 Oct 30 13:03:44.280792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.280773 5693 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:03:44.281146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.281121 5693 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:03:44.331696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.331639 5693 manager.go:324] Recovery completed Oct 30 13:03:44.347145 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.347113 5693 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 13:03:44.355657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355639 5693 cpu_manager.go:226] "Starting" policy="none" Oct 30 13:03:44.355657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355654 5693 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 13:03:44.355784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355670 5693 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 13:03:44.355784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355780 5693 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 13:03:44.355840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355790 5693 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 13:03:44.355840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355807 5693 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 13:03:44.355840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355812 5693 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 13:03:44.355840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355818 5693 policy_none.go:50] "Start" Oct 30 13:03:44.355840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355827 5693 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 13:03:44.355840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355836 5693 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 13:03:44.355996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355962 5693 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 13:03:44.355996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355970 5693 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 13:03:44.355996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.355976 5693 policy_none.go:44] "Start" Oct 30 13:03:44.360480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.360453 5693 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 13:03:44.365737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.365722 5693 manager.go:341] "Starting Device Plugin manager" Oct 30 13:03:44.366300 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.365745 5693 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 13:03:44.366365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366304 5693 server.go:87] "Starting device plugin registration server" Oct 30 13:03:44.366547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366538 5693 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 13:03:44.366580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366548 5693 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 13:03:44.366678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366660 5693 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 13:03:44.366794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366761 5693 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 13:03:44.366852 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366801 5693 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 13:03:44.366852 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366808 5693 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 13:03:44.366948 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.366857 5693 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:03:44.366792007Z","UUID":"12be8076-99cb-4b09-a2fb-18a30da415ff","Handler":null,"Name":"","Endpoint":""} Oct 30 13:03:44.368609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.368561 5693 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 13:03:44.368697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.368647 5693 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 13:03:44.368803 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.368781 5693 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 13:03:44.467569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.467536 5693 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:03:44.469628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.469591 5693 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:03:44.469723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.469639 5693 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:03:44.469723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.469654 5693 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-43.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:03:44.469723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.469707 5693 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.502236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.502208 5693 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.502352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.502283 5693 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.547799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.547740 5693 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal"] Oct 30 13:03:44.548524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.548470 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" event={"ID":"146c4e2eccc94fc30a7e9ef0c46e8801","Type":"ContainerDied","Data":"4f5c6ceff6f29ec7c216d9ef79e526c941363e35e6532491570e786fb20fe62d"} Oct 30 13:03:44.548524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.548517 5693 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.548708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.548531 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" event={"ID":"146c4e2eccc94fc30a7e9ef0c46e8801","Type":"ContainerStarted","Data":"c12cb841620c3ecb0f34c8ef9ab9b81f16d22b4401f444fa53b8d0419260c54f"} Oct 30 13:03:44.548708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.548588 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" event={"ID":"9c6cf73d04b1de3c177c6f664e9eb174","Type":"ContainerStarted","Data":"a1e2081deaeb9dd5a8fd161b6eed4c4a9a32a67db41e8acc71b8cd8ae1eb01db"} Oct 30 13:03:44.548708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.548630 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" event={"ID":"9c6cf73d04b1de3c177c6f664e9eb174","Type":"ContainerStarted","Data":"7774e42fbd558997ad9cc9b27f66a228b787750b82124933aa06aa8f603c5f38"} Oct 30 13:03:44.548813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.548722 5693 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.573421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.573402 5693 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:03:44.573479 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.573441 5693 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.573695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.573683 5693 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:03:44.573728 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:44.573713 5693 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.659017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.658939 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.659017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.658978 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.659017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.659000 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c6cf73d04b1de3c177c6f664e9eb174-config\") pod \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" (UID: \"9c6cf73d04b1de3c177c6f664e9eb174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.759542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.759513 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.759542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.759544 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.759733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.759563 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c6cf73d04b1de3c177c6f664e9eb174-config\") pod \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" (UID: \"9c6cf73d04b1de3c177c6f664e9eb174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.759733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.759649 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.759733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.759678 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c6cf73d04b1de3c177c6f664e9eb174-config\") pod \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" (UID: \"9c6cf73d04b1de3c177c6f664e9eb174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:44.759733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:44.759650 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/146c4e2eccc94fc30a7e9ef0c46e8801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" (UID: \"146c4e2eccc94fc30a7e9ef0c46e8801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:45.198257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.198223 5693 apiserver.go:52] "Watching apiserver" Oct 30 13:03:45.250091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.250060 5693 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 13:03:45.250751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.250724 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79cdc67ddb-bx2c7","openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9","openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx","openshift-network-diagnostics/network-check-target-gfvdc","openshift-dns/node-resolver-gd79z","openshift-multus/multus-qq4pk","kube-system/konnectivity-agent-7zncw","openshift-monitoring/telemeter-client-549bbd8b7d-lf666","openshift-console/console-d94544d6f-wrh5c","openshift-console/console-5d84dd8f56-zc48q","openshift-dns/dns-default-dg69d","kube-system/global-pull-secret-syncer-kn246","openshift-cluster-node-tuning-operator/tuned-nsdcz","openshift-ingress-canary/ingress-canary-5zhld","openshift-ovn-kubernetes/ovnkube-node-w6wcv","openshift-insights/insights-runtime-extractor-6zw7d","kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks","openshift-monitoring/node-exporter-qdws6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt","openshift-multus/network-metrics-daemon-hjjpd","openshift-console/console-857c87f86-l2ddg","openshift-image-registry/node-ca-8kg5l","openshift-multus/multus-additional-cni-plugins-jd79k","openshift-network-operator/iptables-alerter-f696x","openshift-console/console-86dc676d59-98t6x"] Oct 30 13:03:45.267068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267041 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 13:03:45.267225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267077 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.267225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267099 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.267225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267184 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-prxcs\"" Oct 30 13:03:45.267370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267291 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.267559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267534 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.267824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267802 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 13:03:45.267979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267961 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.268044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.267977 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.268044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268036 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 13:03:45.268044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268040 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268092 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268095 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268036 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268097 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268147 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268171 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxh59\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268173 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.268205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268202 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 13:03:45.268502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268229 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vz5h4\"" Oct 30 13:03:45.268502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268274 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r278v\"" Oct 30 13:03:45.268502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268375 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8jrt\"" Oct 30 13:03:45.268502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.268374 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrlh9\"" Oct 30 13:03:45.269087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.269065 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.276488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.276463 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 13:03:45.276580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.276465 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.276580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.276464 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.276580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.276520 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 13:03:45.276580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.276543 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.276732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.276687 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxrfm\"" Oct 30 13:03:45.278343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.278329 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 13:03:45.278590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.278574 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 13:03:45.279164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.279152 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 13:03:45.315473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.315444 5693 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 13:03:45.318699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.318676 5693 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:45.318829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.318750 5693 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:45.338895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.338868 5693 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:03:45.339062 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:45.339014 5693 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-43.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-43.ec2.internal" Oct 30 13:03:45.339124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.339108 5693 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:03:45.339158 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:45.339145 5693 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-43.ec2.internal" Oct 30 13:03:45.361469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361443 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.361568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361474 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-sys-fs\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.361568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361493 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-dbus\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.361568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361517 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdfaae46-aa7b-4311-8ea5-1f6716311a36-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.361694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361567 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.361694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361628 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdfaae46-aa7b-4311-8ea5-1f6716311a36-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.361757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.361757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-slash\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.361757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361746 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.361844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361764 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.361844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361821 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.361910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361855 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-bin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.361910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-run\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.361978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361907 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da949e5b-eb60-4823-b417-41067e8d0959-config-volume\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.361978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361930 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysconfig\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.361978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361953 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-serviceca\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.362079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.361978 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.362079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362003 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.362079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362030 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfqr\" (UniqueName: \"kubernetes.io/projected/a56d5807-0ea6-48b5-8487-64330925498a-kube-api-access-6wfqr\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.362079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362072 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362059 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362108 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec11257-4c9d-48b9-9333-970360865ea7-tmp-dir\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362119 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da949e5b-eb60-4823-b417-41067e8d0959-config-volume\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362135 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-netns\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362152 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-root\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362173 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.362231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362213 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362246 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwm5n\" (UniqueName: \"kubernetes.io/projected/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-kube-api-access-hwm5n\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362245 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec11257-4c9d-48b9-9333-970360865ea7-tmp-dir\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362273 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362334 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-k8s-cni-cncf-io\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-metrics-server-audit-profiles\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362402 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbcd\" (UniqueName: \"kubernetes.io/projected/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-api-access-7jbcd\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362430 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362456 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-multus\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.362489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362483 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-hostroot\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362509 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362647 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362701 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-sys\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362740 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362740 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-serviceca\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362811 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbkp\" (UniqueName: \"kubernetes.io/projected/34b15028-7cbf-46f3-9ffd-d9a8369128af-kube-api-access-zfbkp\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362852 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fec11257-4c9d-48b9-9333-970360865ea7-hosts-file\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.362884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362876 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-system-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362921 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-kubelet\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.362955 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-netns\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363017 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjlz\" (UniqueName: \"kubernetes.io/projected/134089b3-8c7a-4f4b-8330-127fb60ba056-kube-api-access-qbjlz\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363053 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-metrics-client-ca\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363079 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363134 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363162 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-env-overrides\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363190 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-script-lib\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.363252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363216 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363274 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363370 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-lib-modules\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363406 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4f4\" (UniqueName: \"kubernetes.io/projected/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-kube-api-access-pw4f4\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363405 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363453 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-env-overrides\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363460 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-crio-socket\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363511 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363540 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-node-log\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363565 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-device-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-data-volume\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.363688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363684 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-data-volume\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363718 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363746 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-netd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363771 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-tmp\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363799 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363867 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363947 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363966 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363975 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-conf-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-tmp\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.363996 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364000 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-multus-certs\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364028 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7db\" (UniqueName: \"kubernetes.io/projected/d3cd14b5-9230-48f3-8f92-67425d7a0122-kube-api-access-cm7db\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364064 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da949e5b-eb60-4823-b417-41067e8d0959-tmp-dir\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364076 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.364179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364162 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-metrics-client-ca\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364194 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364211 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da949e5b-eb60-4823-b417-41067e8d0959-tmp-dir\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364219 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6bs\" (UniqueName: \"kubernetes.io/projected/19c122c0-62b5-437d-91d5-3d519c7958f7-kube-api-access-bp6bs\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364254 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a085e4-85e9-466c-984c-e08ccc0b8862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364278 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364298 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364360 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-kubelet\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-socket-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364432 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da949e5b-eb60-4823-b417-41067e8d0959-metrics-tls\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364463 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-konnectivity-ca\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364489 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364683 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364715 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovn-node-metrics-cert\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364743 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/187fc60f-7b2a-45dd-b5ff-13839099e07f-audit-log\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364767 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npntf\" (UniqueName: \"kubernetes.io/projected/187fc60f-7b2a-45dd-b5ff-13839099e07f-kube-api-access-npntf\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364791 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134089b3-8c7a-4f4b-8330-127fb60ba056-host-slash\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364816 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364834 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364849 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364865 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nzl\" (UniqueName: \"kubernetes.io/projected/e060d933-f69d-44fd-a416-6a9877c0cfed-kube-api-access-j6nzl\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364880 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-kubelet-config\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364900 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364916 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.364924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364932 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364947 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364962 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364978 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-socket-dir-parent\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.364993 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-multus-daemon-config\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365007 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-ovn\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365024 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-client-certs\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365042 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365063 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-var-lib-kubelet\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365083 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-host\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365107 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblw7\" (UniqueName: \"kubernetes.io/projected/05e9b6ca-1d68-43eb-a809-24debe93ae5b-kube-api-access-qblw7\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-system-cni-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365157 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365181 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-cnibin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365206 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-systemd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365223 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8zj\" (UniqueName: \"kubernetes.io/projected/fbc1dd68-9c18-40cf-9d70-c812d51564fc-kube-api-access-nm8zj\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365239 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134089b3-8c7a-4f4b-8330-127fb60ba056-iptables-alerter-script\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365255 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-registration-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365269 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-etc-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365284 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-modprobe-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365299 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-conf\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365316 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-tls\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365339 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-sys\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365356 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365377 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8jb\" (UniqueName: \"kubernetes.io/projected/fec11257-4c9d-48b9-9333-970360865ea7-kube-api-access-dh8jb\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365392 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365409 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365425 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85daeb3-1320-4a20-b831-0817f651f8e8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365441 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-client-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365468 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365487 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365502 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365517 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-tuned\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365533 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365551 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkc7\" (UniqueName: \"kubernetes.io/projected/e85daeb3-1320-4a20-b831-0817f651f8e8-kube-api-access-hbkc7\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365567 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-etc-kubernetes\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365583 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsqn\" (UniqueName: \"kubernetes.io/projected/48592583-022f-490d-b547-a4876a43da65-kube-api-access-dcsqn\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.365590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365614 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-log-socket\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365640 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-config\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365694 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365773 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-konnectivity-ca\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365801 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365851 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-tuned\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365908 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/187fc60f-7b2a-45dd-b5ff-13839099e07f-audit-log\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365931 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.365949 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-textfile\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366040 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-multus-daemon-config\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366058 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-textfile\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366130 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134089b3-8c7a-4f4b-8330-127fb60ba056-iptables-alerter-script\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366134 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366161 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-wtmp\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366297 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-cnibin\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366329 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-systemd-units\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366378 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-systemd\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366411 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-host\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366460 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366482 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-serving-certs-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366535 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366578 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e9b6ca-1d68-43eb-a809-24debe93ae5b-cert\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366680 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-kubernetes\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366761 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366778 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366792 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366836 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366871 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366897 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-tls\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366939 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366943 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366976 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.366981 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367009 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpr7\" (UniqueName: \"kubernetes.io/projected/78a085e4-85e9-466c-984c-e08ccc0b8862-kube-api-access-pgpr7\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367101 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.367123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367193 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367218 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tcpr\" (UniqueName: \"kubernetes.io/projected/da949e5b-eb60-4823-b417-41067e8d0959-kube-api-access-5tcpr\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367302 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367308 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367302 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367345 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g884v\" (UniqueName: \"kubernetes.io/projected/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-api-access-g884v\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367355 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367408 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367519 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tdfvc\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367518 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8w9b7\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367580 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27z9n\" (UniqueName: \"kubernetes.io/projected/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-kube-api-access-27z9n\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-agent-certs\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367753 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367758 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367777 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367790 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-cni-binary-copy\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367825 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367826 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367828 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pbv95\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367793 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367871 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.367888 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368028 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48592583-022f-490d-b547-a4876a43da65-cni-binary-copy\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368062 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-bin\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368103 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368132 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-federate-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368146 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aaa6be8-1e4d-44a0-91af-6a98718ba6e8-agent-certs\") pod \"konnectivity-agent-7zncw\" (UID: \"0aaa6be8-1e4d-44a0-91af-6a98718ba6e8\") " pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368180 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368208 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-var-lib-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368227 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368279 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368311 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368278 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368328 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368315 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbfrs\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368354 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368434 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-federate-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368486 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwzrd\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368508 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368518 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368518 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fx9p2\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368523 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368540 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368563 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368563 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368613 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 13:03:45.368689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368665 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368830 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368840 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hbwxh\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368860 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368853 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368865 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368906 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368920 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368925 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368929 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368938 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368945 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368952 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368967 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368971 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qckc2\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368980 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368982 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.368981 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-467j9\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.369014 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9dqpm\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.369058 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.369162 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.369207 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4gwc5\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.369283 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 13:03:45.370093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.369312 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\"" Oct 30 13:03:45.372441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.372416 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.372546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.372437 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-client-tls\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.372747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.372722 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-metrics-server-audit-profiles\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.372868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.372837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.372913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.372901 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.373620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.373581 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.373964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.373946 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.374096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.374077 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a56d5807-0ea6-48b5-8487-64330925498a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.374775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.374748 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.374775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.374765 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.374905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.374829 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da949e5b-eb60-4823-b417-41067e8d0959-metrics-tls\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376060 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3cd14b5-9230-48f3-8f92-67425d7a0122-metrics-certs\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376070 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376102 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376085 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376113 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/78a085e4-85e9-466c-984c-e08ccc0b8862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376224 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.376298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376248 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-client-certs\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376348 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376369 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19c122c0-62b5-437d-91d5-3d519c7958f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376257 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376514 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovn-node-metrics-cert\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376556 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-config\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376702 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-secret-metrics-server-tls\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376732 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-serving-certs-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.376790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376770 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.377072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.376959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e9b6ca-1d68-43eb-a809-24debe93ae5b-cert\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:45.377304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.377282 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.377379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.377320 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-tls\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.377416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.377392 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-original-pull-secret\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.378006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.377991 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.378800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.378786 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187fc60f-7b2a-45dd-b5ff-13839099e07f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379054 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379088 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379094 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379112 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379145 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pjfzd\"" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379097 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2jm058lbepfmq\"" Oct 30 13:03:45.379259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379237 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mk5rw\"" Oct 30 13:03:45.379723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.379288 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 13:03:45.380097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.380077 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 13:03:45.383479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.382032 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.383479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.382885 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbkp\" (UniqueName: \"kubernetes.io/projected/34b15028-7cbf-46f3-9ffd-d9a8369128af-kube-api-access-zfbkp\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.383479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.382949 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.383702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.383567 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") pod \"console-86dc676d59-98t6x\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.384197 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbc1dd68-9c18-40cf-9d70-c812d51564fc-ovnkube-script-lib\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.384306 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwm5n\" (UniqueName: \"kubernetes.io/projected/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-kube-api-access-hwm5n\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.384419 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.384646 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a085e4-85e9-466c-984c-e08ccc0b8862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.384741 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npntf\" (UniqueName: \"kubernetes.io/projected/187fc60f-7b2a-45dd-b5ff-13839099e07f-kube-api-access-npntf\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.385256 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfqr\" (UniqueName: \"kubernetes.io/projected/a56d5807-0ea6-48b5-8487-64330925498a-kube-api-access-6wfqr\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.385312 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjlz\" (UniqueName: \"kubernetes.io/projected/134089b3-8c7a-4f4b-8330-127fb60ba056-kube-api-access-qbjlz\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.385450 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7db\" (UniqueName: \"kubernetes.io/projected/d3cd14b5-9230-48f3-8f92-67425d7a0122-kube-api-access-cm7db\") pod \"network-metrics-daemon-hjjpd\" (UID: \"d3cd14b5-9230-48f3-8f92-67425d7a0122\") " pod="openshift-multus/network-metrics-daemon-hjjpd" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.385843 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-metrics-client-ca\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386035 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-metrics-client-ca\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386062 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e86df5d-83ed-433e-9c0e-b5f59970dfe9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mdb6n\" (UID: \"6e86df5d-83ed-433e-9c0e-b5f59970dfe9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mdb6n" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386057 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6bs\" (UniqueName: \"kubernetes.io/projected/19c122c0-62b5-437d-91d5-3d519c7958f7-kube-api-access-bp6bs\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386141 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386301 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbcd\" (UniqueName: \"kubernetes.io/projected/bdfaae46-aa7b-4311-8ea5-1f6716311a36-kube-api-access-7jbcd\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386366 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85daeb3-1320-4a20-b831-0817f651f8e8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386433 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfaae46-aa7b-4311-8ea5-1f6716311a36-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t87s9\" (UID: \"bdfaae46-aa7b-4311-8ea5-1f6716311a36\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t87s9" Oct 30 13:03:45.387008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.386784 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/187fc60f-7b2a-45dd-b5ff-13839099e07f-client-ca-bundle\") pod \"metrics-server-79cdc67ddb-bx2c7\" (UID: \"187fc60f-7b2a-45dd-b5ff-13839099e07f\") " pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:03:45.387927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.387348 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.388097 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.388493 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") pod \"console-5d84dd8f56-zc48q\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.388979 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.388991 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.389131 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblw7\" (UniqueName: \"kubernetes.io/projected/05e9b6ca-1d68-43eb-a809-24debe93ae5b-kube-api-access-qblw7\") pod \"ingress-canary-5zhld\" (UID: \"05e9b6ca-1d68-43eb-a809-24debe93ae5b\") " pod="openshift-ingress-canary/ingress-canary-5zhld" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.389564 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27z9n\" (UniqueName: \"kubernetes.io/projected/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-kube-api-access-27z9n\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.389662 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8jb\" (UniqueName: \"kubernetes.io/projected/fec11257-4c9d-48b9-9333-970360865ea7-kube-api-access-dh8jb\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.389813 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpr7\" (UniqueName: \"kubernetes.io/projected/78a085e4-85e9-466c-984c-e08ccc0b8862-kube-api-access-pgpr7\") pod \"prometheus-operator-9cd89b74c-8vmwx\" (UID: \"78a085e4-85e9-466c-984c-e08ccc0b8862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vmwx" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.389888 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbv2m\" (UniqueName: \"kubernetes.io/projected/89c9725b-8217-433e-8d03-d8a493c9f336-kube-api-access-mbv2m\") pod \"network-check-target-gfvdc\" (UID: \"89c9725b-8217-433e-8d03-d8a493c9f336\") " pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390201 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tcpr\" (UniqueName: \"kubernetes.io/projected/da949e5b-eb60-4823-b417-41067e8d0959-kube-api-access-5tcpr\") pod \"dns-default-dg69d\" (UID: \"da949e5b-eb60-4823-b417-41067e8d0959\") " pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390232 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8zj\" (UniqueName: \"kubernetes.io/projected/fbc1dd68-9c18-40cf-9d70-c812d51564fc-kube-api-access-nm8zj\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") pod \"console-857c87f86-l2ddg\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390575 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g884v\" (UniqueName: \"kubernetes.io/projected/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-kube-api-access-g884v\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nzl\" (UniqueName: \"kubernetes.io/projected/e060d933-f69d-44fd-a416-6a9877c0cfed-kube-api-access-j6nzl\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390716 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkc7\" (UniqueName: \"kubernetes.io/projected/e85daeb3-1320-4a20-b831-0817f651f8e8-kube-api-access-hbkc7\") pod \"monitoring-plugin-77f4f76594-d5hqt\" (UID: \"e85daeb3-1320-4a20-b831-0817f651f8e8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:45.391109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.390782 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsqn\" (UniqueName: \"kubernetes.io/projected/48592583-022f-490d-b547-a4876a43da65-kube-api-access-dcsqn\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.391710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.391373 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4f4\" (UniqueName: \"kubernetes.io/projected/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-kube-api-access-pw4f4\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.395732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.395711 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") pod \"console-d94544d6f-wrh5c\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:03:45.396958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.396940 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56d5807-0ea6-48b5-8487-64330925498a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549bbd8b7d-lf666\" (UID: \"a56d5807-0ea6-48b5-8487-64330925498a\") " pod="openshift-monitoring/telemeter-client-549bbd8b7d-lf666" Oct 30 13:03:45.398033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.398012 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857c87f86-l2ddg"] Oct 30 13:03:45.468931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468851 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-run\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.468931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysconfig\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.468931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468911 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-netns\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.468931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468932 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-root\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468971 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-run\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468978 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-root\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468985 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-netns\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.468988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysconfig\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469021 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-k8s-cni-cncf-io\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469057 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-multus\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-k8s-cni-cncf-io\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469080 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-hostroot\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469109 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469113 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-hostroot\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469129 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469142 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-multus\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469158 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469154 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-sys\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-sys\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469202 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469222 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469231 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fec11257-4c9d-48b9-9333-970360865ea7-hosts-file\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469257 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-system-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469292 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fec11257-4c9d-48b9-9333-970360865ea7-hosts-file\") pod \"node-resolver-gd79z\" (UID: \"fec11257-4c9d-48b9-9333-970360865ea7\") " pod="openshift-dns/node-resolver-gd79z" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469296 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-kubelet\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469312 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-system-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469316 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469320 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-netns\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-run-netns\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469324 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-kubelet\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469373 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469402 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-lib-modules\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469428 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-crio-socket\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469452 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-node-log\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469476 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-device-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469497 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469506 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-node-log\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469502 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-lib-modules\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469502 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-netd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469543 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-device-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469553 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-conf-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469499 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc-crio-socket\") pod \"insights-runtime-extractor-6zw7d\" (UID: \"0f2f8ea2-5dbd-4cf5-8be8-d725dbb7e3fc\") " pod="openshift-insights/insights-runtime-extractor-6zw7d" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469583 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-multus-certs\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469590 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-netd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469589 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-conf-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469628 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-run-multus-certs\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-kubelet\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469659 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-socket-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469660 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-kubelet\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469679 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134089b3-8c7a-4f4b-8330-127fb60ba056-host-slash\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469702 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-kubelet-config\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469724 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-socket-dir-parent\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469735 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-socket-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469740 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-ovn\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469763 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134089b3-8c7a-4f4b-8330-127fb60ba056-host-slash\") pod \"iptables-alerter-f696x\" (UID: \"134089b3-8c7a-4f4b-8330-127fb60ba056\") " pod="openshift-network-operator/iptables-alerter-f696x" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-ovn\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469799 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-kubelet-config\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-socket-dir-parent\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469841 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469842 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-var-lib-kubelet\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469878 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-var-lib-kubelet\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.469876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469885 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-host\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469909 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-system-cni-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469925 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-cnibin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469927 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-host\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469944 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-system-cni-dir\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469943 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-systemd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-cnibin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469976 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-systemd\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.469984 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-registration-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470010 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-etc-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470036 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-modprobe-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470059 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-conf\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470066 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-registration-dir\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470084 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-sys\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470109 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470114 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-sys\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470085 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-etc-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470121 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-modprobe-d\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470142 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-etc-kubernetes\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470152 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-sysctl-conf\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470166 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-log-socket\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470175 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-etc-kubernetes\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470183 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-wtmp\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470185 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-multus-cni-dir\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470214 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-log-socket\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470208 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-cnibin\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470235 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ca8a3b3-0f1e-4bd2-8660-6d8add527517-node-exporter-wtmp\") pod \"node-exporter-qdws6\" (UID: \"2ca8a3b3-0f1e-4bd2-8660-6d8add527517\") " pod="openshift-monitoring/node-exporter-qdws6" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470241 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19c122c0-62b5-437d-91d5-3d519c7958f7-cnibin\") pod \"multus-additional-cni-plugins-jd79k\" (UID: \"19c122c0-62b5-437d-91d5-3d519c7958f7\") " pod="openshift-multus/multus-additional-cni-plugins-jd79k" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470248 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-systemd-units\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-systemd\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470293 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-host\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470314 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-systemd-units\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5276b8dc-f3cf-4d3b-a630-bd16aca9a373-host\") pod \"node-ca-8kg5l\" (UID: \"5276b8dc-f3cf-4d3b-a630-bd16aca9a373\") " pod="openshift-image-registry/node-ca-8kg5l" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470341 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-systemd\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470341 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-kubernetes\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470380 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e060d933-f69d-44fd-a416-6a9877c0cfed-etc-kubernetes\") pod \"tuned-nsdcz\" (UID: \"e060d933-f69d-44fd-a416-6a9877c0cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-nsdcz" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470388 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-bin\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470415 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-var-lib-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470439 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470446 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-var-lib-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470418 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-cni-bin\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470463 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-sys-fs\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470494 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-run-openvswitch\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470502 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-dbus\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470534 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-slash\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470542 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-sys-fs\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470559 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbc1dd68-9c18-40cf-9d70-c812d51564fc-host-slash\") pod \"ovnkube-node-w6wcv\" (UID: \"fbc1dd68-9c18-40cf-9d70-c812d51564fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470615 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6aaafde-99f7-4cdb-9c95-381d18e5f1b1-dbus\") pod \"global-pull-secret-syncer-kn246\" (UID: \"d6aaafde-99f7-4cdb-9c95-381d18e5f1b1\") " pod="kube-system/global-pull-secret-syncer-kn246" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470591 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-bin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470635 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b15028-7cbf-46f3-9ffd-d9a8369128af-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzmks\" (UID: \"34b15028-7cbf-46f3-9ffd-d9a8369128af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzmks" Oct 30 13:03:45.471203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:45.470642 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48592583-022f-490d-b547-a4876a43da65-host-var-lib-cni-bin\") pod \"multus-qq4pk\" (UID: \"48592583-022f-490d-b547-a4876a43da65\") " pod="openshift-multus/multus-qq4pk" Oct 30 13:03:46.567548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:46.567519 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dg69d" Oct 30 13:03:46.867920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:46.867851 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:46.876578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:46.876557 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:03:50.745458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:50.745422 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:50.761945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:50.761915 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:03:50.900789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:50.900748 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:50.931799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:50.931776 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:03:50.956295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:50.956264 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d5hqt" Oct 30 13:03:51.022955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:51.022881 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:51.023106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:51.023073 5693 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:03:51.093791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:51.093762 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7zncw" Oct 30 13:03:51.320964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:51.320885 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.450663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:51.450628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:52.725488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:52.725458 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6wcv" Oct 30 13:03:53.053868 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:03:53.053789 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:03:54.764325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:03:54.764293 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:04:00.936661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:00.936626 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:04:00.940321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:00.940302 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:04:01.329999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:01.329963 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.459500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:01.459463 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:02.564401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:02.564361 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:02.696933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:02.696903 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.752233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:07.752202 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79cdc67ddb-bx2c7" Oct 30 13:04:08.435928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:08.435895 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:08.575818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:08.575783 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:10.278287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.278240 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86dc676d59-98t6x" podUID="5ad840e4-950d-4761-a526-65c645e7de03" containerName="console" containerID="cri-o://657a2b2dd89c5454eee322cc091dcea656f8f0c457810707dbdca8666ade3b92" gracePeriod=15 Oct 30 13:04:10.281918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.281885 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d94544d6f-wrh5c" podUID="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" containerName="console" containerID="cri-o://3c80e9be7b7f5d5e27eb7c757432ef53857ef25039787bbdf344c7950304928a" gracePeriod=15 Oct 30 13:04:10.409676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.409645 5693 generic.go:356] "Generic (PLEG): container finished" podID="5ad840e4-950d-4761-a526-65c645e7de03" containerID="657a2b2dd89c5454eee322cc091dcea656f8f0c457810707dbdca8666ade3b92" exitCode=0 Oct 30 13:04:10.409812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.409733 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc676d59-98t6x" event={"ID":"5ad840e4-950d-4761-a526-65c645e7de03","Type":"ContainerDied","Data":"657a2b2dd89c5454eee322cc091dcea656f8f0c457810707dbdca8666ade3b92"} Oct 30 13:04:10.411313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.411291 5693 generic.go:356] "Generic (PLEG): container finished" podID="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" containerID="3c80e9be7b7f5d5e27eb7c757432ef53857ef25039787bbdf344c7950304928a" exitCode=0 Oct 30 13:04:10.411425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.411361 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d94544d6f-wrh5c" event={"ID":"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5","Type":"ContainerDied","Data":"3c80e9be7b7f5d5e27eb7c757432ef53857ef25039787bbdf344c7950304928a"} Oct 30 13:04:10.544975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.544948 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:04:10.547726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.547706 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:04:10.584568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584541 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.584735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584589 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.584735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584630 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.584735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584669 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.584735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584706 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584743 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584784 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584817 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584842 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584872 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584900 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.584945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584927 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.585268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584959 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") pod \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\" (UID: \"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5\") " Oct 30 13:04:10.585268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.584983 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.585268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.585027 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") pod \"5ad840e4-950d-4761-a526-65c645e7de03\" (UID: \"5ad840e4-950d-4761-a526-65c645e7de03\") " Oct 30 13:04:10.586240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.586204 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.586498 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.586528 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.586780 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.586897 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587336 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587548 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587637 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587693 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587695 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:10.587799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587786 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:10.588274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.587950 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:10.589547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.589525 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "kube-api-access-v2jtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:10.589780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.589738 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert" pod "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" (UID: "6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:10.589856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.589832 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl" pod "5ad840e4-950d-4761-a526-65c645e7de03" (UID: "5ad840e4-950d-4761-a526-65c645e7de03"). InnerVolumeSpecName "kube-api-access-mm2pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:10.685813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685778 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.685813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685807 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ad840e4-950d-4761-a526-65c645e7de03-console-oauth-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.685813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685819 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685830 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685838 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-console-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685847 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-oauth-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685855 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm2pl\" (UniqueName: \"kubernetes.io/projected/5ad840e4-950d-4761-a526-65c645e7de03-kube-api-access-mm2pl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685866 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad840e4-950d-4761-a526-65c645e7de03-tmp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685876 5693 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-trusted-ca-bundle\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685885 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685894 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-oauth-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685901 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-tmp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685910 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jtv\" (UniqueName: \"kubernetes.io/projected/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-kube-api-access-v2jtv\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685917 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5-console-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.686038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.685926 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ad840e4-950d-4761-a526-65c645e7de03-oauth-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:10.760790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.760762 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-545cc6f55f-j7g67"] Oct 30 13:04:10.761049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.761021 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" containerName="console" Oct 30 13:04:10.761049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.761038 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" containerName="console" Oct 30 13:04:10.761049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.761062 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5ad840e4-950d-4761-a526-65c645e7de03" containerName="console" Oct 30 13:04:10.761287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.761069 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad840e4-950d-4761-a526-65c645e7de03" containerName="console" Oct 30 13:04:10.761287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.761231 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad840e4-950d-4761-a526-65c645e7de03" containerName="console" Oct 30 13:04:10.761287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.761252 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" containerName="console" Oct 30 13:04:10.765719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.765699 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.785628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.785556 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545cc6f55f-j7g67"] Oct 30 13:04:10.887739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.887702 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkzn\" (UniqueName: \"kubernetes.io/projected/8174a28b-9f7a-4bde-942d-08e72cc5b097-kube-api-access-gwkzn\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.887739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.887741 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-oauth-serving-cert\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.887944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.887766 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-trusted-ca-bundle\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.887944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.887901 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-oauth-config\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.887944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.887929 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8174a28b-9f7a-4bde-942d-08e72cc5b097-tmp\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.888036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.887988 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-serving-cert\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.888070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.888039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-config\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.888108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.888089 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-service-ca\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.938272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.938245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:10.989000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.988971 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkzn\" (UniqueName: \"kubernetes.io/projected/8174a28b-9f7a-4bde-942d-08e72cc5b097-kube-api-access-gwkzn\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989006 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-oauth-serving-cert\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989026 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-trusted-ca-bundle\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989068 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-oauth-config\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989083 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8174a28b-9f7a-4bde-942d-08e72cc5b097-tmp\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989125 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-serving-cert\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989152 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-config\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989199 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-service-ca\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.989644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989573 5693 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 13:04:10.989644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.989590 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8174a28b-9f7a-4bde-942d-08e72cc5b097-tmp\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.990134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.990095 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-oauth-serving-cert\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.990241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.990172 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-service-ca\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.990241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.990181 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-config\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.990674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.990654 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174a28b-9f7a-4bde-942d-08e72cc5b097-trusted-ca-bundle\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.991891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.991870 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-oauth-config\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:10.991978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:10.991954 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8174a28b-9f7a-4bde-942d-08e72cc5b097-console-serving-cert\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:11.008039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.008016 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkzn\" (UniqueName: \"kubernetes.io/projected/8174a28b-9f7a-4bde-942d-08e72cc5b097-kube-api-access-gwkzn\") pod \"console-545cc6f55f-j7g67\" (UID: \"8174a28b-9f7a-4bde-942d-08e72cc5b097\") " pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:11.077027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.076955 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:11.083185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.083164 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.339689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.339614 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.340943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.340915 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-857c87f86-l2ddg" podUID="fed238e7-3fbe-47d0-8748-eafbed34a837" containerName="console" containerID="cri-o://b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7" gracePeriod=15 Oct 30 13:04:11.371415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.371389 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545cc6f55f-j7g67"] Oct 30 13:04:11.400765 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:04:11.400725 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8174a28b_9f7a_4bde_942d_08e72cc5b097.slice/crio-c0a091e8db56df057f6612256f8876dc596be09641f859eb5b687f2b5ff84a68 WatchSource:0}: Error finding container c0a091e8db56df057f6612256f8876dc596be09641f859eb5b687f2b5ff84a68: Status 404 returned error can't find the container with id c0a091e8db56df057f6612256f8876dc596be09641f859eb5b687f2b5ff84a68 Oct 30 13:04:11.416285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.416248 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc676d59-98t6x" event={"ID":"5ad840e4-950d-4761-a526-65c645e7de03","Type":"ContainerDied","Data":"73aaa0154c6f603316a16884183922e64edd98c515e0f32f31530487ff4feecb"} Oct 30 13:04:11.416437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.416334 5693 scope.go:122] "RemoveContainer" containerID="657a2b2dd89c5454eee322cc091dcea656f8f0c457810707dbdca8666ade3b92" Oct 30 13:04:11.416437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.416264 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc676d59-98t6x" Oct 30 13:04:11.417733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.417717 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d94544d6f-wrh5c" Oct 30 13:04:11.417817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.417715 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d94544d6f-wrh5c" event={"ID":"6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5","Type":"ContainerDied","Data":"5c1108eb9fb8022ac81a83699c657d89599ecd0367f742cf5073011b2a6e7a54"} Oct 30 13:04:11.451328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.451305 5693 scope.go:122] "RemoveContainer" containerID="3c80e9be7b7f5d5e27eb7c757432ef53857ef25039787bbdf344c7950304928a" Oct 30 13:04:11.466347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.466299 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d94544d6f-wrh5c"] Oct 30 13:04:11.476570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.476548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.477926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.477904 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d94544d6f-wrh5c"] Oct 30 13:04:11.505249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.505220 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dc676d59-98t6x"] Oct 30 13:04:11.508995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.508973 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86dc676d59-98t6x"] Oct 30 13:04:11.579443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.579419 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:04:11.695728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.695647 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.695728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.695708 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.695938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.695747 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.695938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.695781 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.695938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.695807 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.696085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.695958 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.696085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696011 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.696085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696043 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") pod \"fed238e7-3fbe-47d0-8748-eafbed34a837\" (UID: \"fed238e7-3fbe-47d0-8748-eafbed34a837\") " Oct 30 13:04:11.696239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696135 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:11.696295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696235 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:11.696429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696406 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fed238e7-3fbe-47d0-8748-eafbed34a837-tmp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.696509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696433 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-oauth-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.696509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696457 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:11.696509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696475 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:11.696695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.696677 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:11.698082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.698064 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:11.698363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.698342 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:11.698436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.698346 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t" pod "fed238e7-3fbe-47d0-8748-eafbed34a837" (UID: "fed238e7-3fbe-47d0-8748-eafbed34a837"). InnerVolumeSpecName "kube-api-access-2pt8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:11.797712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.797672 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-console-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.797712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.797704 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.797712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.797714 5693 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed238e7-3fbe-47d0-8748-eafbed34a837-trusted-ca-bundle\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.797942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.797724 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.797942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.797733 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt8t\" (UniqueName: \"kubernetes.io/projected/fed238e7-3fbe-47d0-8748-eafbed34a837-kube-api-access-2pt8t\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:11.797942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:11.797743 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fed238e7-3fbe-47d0-8748-eafbed34a837-console-oauth-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:12.103078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.103041 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:12.251772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.251738 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad840e4-950d-4761-a526-65c645e7de03" path="/var/lib/kubelet/pods/5ad840e4-950d-4761-a526-65c645e7de03/volumes" Oct 30 13:04:12.252153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.252140 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5" path="/var/lib/kubelet/pods/6bd22a5a-4f7f-41c1-b11c-d4ec7dc781a5/volumes" Oct 30 13:04:12.264118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.264095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:12.423837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.423737 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545cc6f55f-j7g67" event={"ID":"8174a28b-9f7a-4bde-942d-08e72cc5b097","Type":"ContainerStarted","Data":"df1cf9e41d1f36fe27653c981d903c50f083b027600ea10531538ee6eb97a32e"} Oct 30 13:04:12.423837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.423798 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545cc6f55f-j7g67" event={"ID":"8174a28b-9f7a-4bde-942d-08e72cc5b097","Type":"ContainerStarted","Data":"c0a091e8db56df057f6612256f8876dc596be09641f859eb5b687f2b5ff84a68"} Oct 30 13:04:12.426059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.426031 5693 generic.go:356] "Generic (PLEG): container finished" podID="fed238e7-3fbe-47d0-8748-eafbed34a837" containerID="b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7" exitCode=0 Oct 30 13:04:12.426175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.426071 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c87f86-l2ddg" Oct 30 13:04:12.426175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.426071 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c87f86-l2ddg" event={"ID":"fed238e7-3fbe-47d0-8748-eafbed34a837","Type":"ContainerDied","Data":"b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7"} Oct 30 13:04:12.426257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.426201 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c87f86-l2ddg" event={"ID":"fed238e7-3fbe-47d0-8748-eafbed34a837","Type":"ContainerDied","Data":"cbbe48d37341e7dcfaf265a68f5229b035934b5a705b669bb8b7f3adc299c73d"} Oct 30 13:04:12.426257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.426226 5693 scope.go:122] "RemoveContainer" containerID="b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7" Oct 30 13:04:12.434162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.434129 5693 scope.go:122] "RemoveContainer" containerID="b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7" Oct 30 13:04:12.434410 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:04:12.434389 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7\": container with ID starting with b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7 not found: ID does not exist" containerID="b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7" Oct 30 13:04:12.434476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.434424 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7"} err="failed to get container status \"b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7\": rpc error: code = NotFound desc = could not find container \"b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7\": container with ID starting with b55472f1ea7a29e3e7984bb7bb9cebada98cac399010c289859d4b8b2c0812e7 not found: ID does not exist" Oct 30 13:04:12.462514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.462472 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-545cc6f55f-j7g67" podStartSLOduration=2.46245972 podStartE2EDuration="2.46245972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:12.462061843 +0000 UTC m=+28.328680034" watchObservedRunningTime="2026-10-30 13:04:12.46245972 +0000 UTC m=+28.329077892" Oct 30 13:04:12.494965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.494933 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857c87f86-l2ddg"] Oct 30 13:04:12.498526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:12.498500 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-857c87f86-l2ddg"] Oct 30 13:04:14.252301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:14.252269 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed238e7-3fbe-47d0-8748-eafbed34a837" path="/var/lib/kubelet/pods/fed238e7-3fbe-47d0-8748-eafbed34a837/volumes" Oct 30 13:04:16.340075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.340029 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:04:16.340556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.340448 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fed238e7-3fbe-47d0-8748-eafbed34a837" containerName="console" Oct 30 13:04:16.340556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.340467 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed238e7-3fbe-47d0-8748-eafbed34a837" containerName="console" Oct 30 13:04:16.340721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.340591 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed238e7-3fbe-47d0-8748-eafbed34a837" containerName="console" Oct 30 13:04:16.345881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.345857 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.360825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.360775 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:04:16.371762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371737 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 13:04:16.371942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371803 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 13:04:16.371942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371808 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 13:04:16.371942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371832 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 13:04:16.372086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371840 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 13:04:16.372086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371860 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 13:04:16.372086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371875 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9ttmx\"" Oct 30 13:04:16.372086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.371914 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 13:04:16.384726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.384705 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 13:04:16.384838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.384821 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 13:04:16.437751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.437893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437764 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b583b0bf-387f-46e7-8380-43fdedfe15e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.437893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437782 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.437893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437806 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.437893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437860 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437956 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b583b0bf-387f-46e7-8380-43fdedfe15e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.437995 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b583b0bf-387f-46e7-8380-43fdedfe15e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.438013 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b583b0bf-387f-46e7-8380-43fdedfe15e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.438063 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.438100 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.438147 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.438164 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b583b0bf-387f-46e7-8380-43fdedfe15e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.438233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.438198 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xl96\" (UniqueName: \"kubernetes.io/projected/b583b0bf-387f-46e7-8380-43fdedfe15e7-kube-api-access-7xl96\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.538984 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.538948 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.538993 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b583b0bf-387f-46e7-8380-43fdedfe15e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539020 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539055 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539083 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539107 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b583b0bf-387f-46e7-8380-43fdedfe15e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539137 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b583b0bf-387f-46e7-8380-43fdedfe15e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539153 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b583b0bf-387f-46e7-8380-43fdedfe15e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539178 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539220 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539295 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b583b0bf-387f-46e7-8380-43fdedfe15e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539337 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xl96\" (UniqueName: \"kubernetes.io/projected/b583b0bf-387f-46e7-8380-43fdedfe15e7-kube-api-access-7xl96\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.539510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.539389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b583b0bf-387f-46e7-8380-43fdedfe15e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.541133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.540229 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b583b0bf-387f-46e7-8380-43fdedfe15e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.541133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.540874 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b583b0bf-387f-46e7-8380-43fdedfe15e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.542408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.542105 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b583b0bf-387f-46e7-8380-43fdedfe15e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.542408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.542324 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.542580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.542560 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.542673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.542584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.542673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.542659 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.542783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.542682 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.543266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.543246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.543385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.543369 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b583b0bf-387f-46e7-8380-43fdedfe15e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.544273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.544251 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b583b0bf-387f-46e7-8380-43fdedfe15e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.558510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.558491 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xl96\" (UniqueName: \"kubernetes.io/projected/b583b0bf-387f-46e7-8380-43fdedfe15e7-kube-api-access-7xl96\") pod \"alertmanager-main-0\" (UID: \"b583b0bf-387f-46e7-8380-43fdedfe15e7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.671960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.671888 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:04:16.879180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:16.879149 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gfvdc" Oct 30 13:04:17.003421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:17.000799 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:04:17.006902 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:04:17.006877 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb583b0bf_387f_46e7_8380_43fdedfe15e7.slice/crio-f43151cf105ba3dbc89da2a55fead8aad7d85625c12d5f0e88c9cf9f8653ae2b WatchSource:0}: Error finding container f43151cf105ba3dbc89da2a55fead8aad7d85625c12d5f0e88c9cf9f8653ae2b: Status 404 returned error can't find the container with id f43151cf105ba3dbc89da2a55fead8aad7d85625c12d5f0e88c9cf9f8653ae2b Oct 30 13:04:17.448560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:17.448469 5693 generic.go:356] "Generic (PLEG): container finished" podID="b583b0bf-387f-46e7-8380-43fdedfe15e7" containerID="ea20dfd05b253200415aee4412c8b857ee0ebd8e5792dc8689c4f296ea315872" exitCode=0 Oct 30 13:04:17.449039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:17.448562 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerDied","Data":"ea20dfd05b253200415aee4412c8b857ee0ebd8e5792dc8689c4f296ea315872"} Oct 30 13:04:17.449039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:17.448666 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"f43151cf105ba3dbc89da2a55fead8aad7d85625c12d5f0e88c9cf9f8653ae2b"} Oct 30 13:04:17.449900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:17.449885 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:04:19.459611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:19.459517 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"52b9db4e94ac005054a4c3c0770ae706e39bf87b8671315b79e8b529bca5ecd6"} Oct 30 13:04:19.459611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:19.459579 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"f7129e0968be01d422a94999c335dafc9a114f8fcbfbee831e56ac30d679b460"} Oct 30 13:04:19.460012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:19.459627 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"60dfb7450aa88ff79ade623833cbd95f1679c91ef259c88de0c5a927ee851b81"} Oct 30 13:04:19.460012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:19.459650 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"0f65d47540221cc2c3ad28a5f8377e94a584372ba1405f133be6c38e10eac5b5"} Oct 30 13:04:19.460012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:19.459673 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"8ea398ec082cde66adb8781501a1a96760febce5149210ed25b8572bb89a51cd"} Oct 30 13:04:20.466510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:20.466469 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b583b0bf-387f-46e7-8380-43fdedfe15e7","Type":"ContainerStarted","Data":"e7817e67272c46c847c6f9335ece3b53f8dcb6f9982882372f4a3905ab3ebc24"} Oct 30 13:04:20.555050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:20.554997 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.412100403 podStartE2EDuration="4.554982256s" totalImagesPullingTime="2.130607546s" totalInitContainerRuntime="12.274307ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:20.542527176 +0000 UTC m=+36.409145350" watchObservedRunningTime="2026-10-30 13:04:20.554982256 +0000 UTC m=+36.421600429" Oct 30 13:04:21.077839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.077803 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:21.077999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.077867 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:21.082616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.082579 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:21.474174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.474149 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-545cc6f55f-j7g67" Oct 30 13:04:21.542032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.542004 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d84dd8f56-zc48q"] Oct 30 13:04:21.716468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.716435 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:21.934620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:21.934522 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:22.063760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:22.063730 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:23.053464 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:04:23.053431 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:04:32.604476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:32.604440 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:32.749999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:32.749964 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:46.561567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.561500 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d84dd8f56-zc48q" podUID="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" containerName="console" containerID="cri-o://0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794" gracePeriod=15 Oct 30 13:04:46.825413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.825354 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:04:46.900731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900697 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.900929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900760 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.900929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900803 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.900929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900841 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.900929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.901158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900937 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.901158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.900970 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.901158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.901021 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") pod \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\" (UID: \"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4\") " Oct 30 13:04:46.901412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.901333 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:46.901488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.901461 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:46.901962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.901844 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:46.901962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.901858 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:46.901962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.901921 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:46.903250 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.903224 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:46.903461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.903438 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:46.903752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:46.903728 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql" pod "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" (UID: "db8fe315-cf83-4d6b-aeaf-25ef603b5ff4"). InnerVolumeSpecName "kube-api-access-g84ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002371 5693 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-trusted-ca-bundle\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002409 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-oauth-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002424 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-tmp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002440 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002454 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-serving-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002466 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-oauth-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002491 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84ql\" (UniqueName: \"kubernetes.io/projected/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-kube-api-access-g84ql\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.002532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.002505 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4-console-config\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.569619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.565115 5693 generic.go:356] "Generic (PLEG): container finished" podID="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" containerID="0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794" exitCode=0 Oct 30 13:04:47.569619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.565189 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d84dd8f56-zc48q" event={"ID":"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4","Type":"ContainerDied","Data":"0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794"} Oct 30 13:04:47.569619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.565242 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d84dd8f56-zc48q" event={"ID":"db8fe315-cf83-4d6b-aeaf-25ef603b5ff4","Type":"ContainerDied","Data":"a63fe62f46baa050d2cd280a25bb40aeba3be7e7f9eb1daa6f8e59b32e5e2bcd"} Oct 30 13:04:47.569619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.565275 5693 scope.go:122] "RemoveContainer" containerID="0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794" Oct 30 13:04:47.569619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.565452 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d84dd8f56-zc48q" Oct 30 13:04:47.593339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.593310 5693 scope.go:122] "RemoveContainer" containerID="0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794" Oct 30 13:04:47.593831 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:04:47.593628 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794\": container with ID starting with 0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794 not found: ID does not exist" containerID="0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794" Oct 30 13:04:47.593831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.593668 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794"} err="failed to get container status \"0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794\": rpc error: code = NotFound desc = could not find container \"0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794\": container with ID starting with 0d4f4815b025cf7fcb7cfa311e349a0176b28cd360454ed709e560b650c94794 not found: ID does not exist" Oct 30 13:04:47.632622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.626058 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d84dd8f56-zc48q"] Oct 30 13:04:47.637526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:47.637500 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d84dd8f56-zc48q"] Oct 30 13:04:48.251711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:48.251676 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" path="/var/lib/kubelet/pods/db8fe315-cf83-4d6b-aeaf-25ef603b5ff4/volumes" Oct 30 13:04:50.828215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:04:50.828183 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.053401 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:04:53.053367 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:05:23.053593 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:05:23.053556 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:05:29.725008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:05:29.724926 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.950006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:05:50.949971 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:53.053446 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:05:53.053407 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:05:54.285656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:05:54.285623 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.425487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:05:54.425456 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:01.362470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:01.362429 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:01.534457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:01.534426 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.459705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.459674 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925"] Oct 30 13:06:02.460158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.459978 5693 cpu_manager.go:414] "containerMap: removing container" podUID="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" containerName="console" Oct 30 13:06:02.460158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.459990 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" containerName="console" Oct 30 13:06:02.460158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.460083 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8fe315-cf83-4d6b-aeaf-25ef603b5ff4" containerName="console" Oct 30 13:06:02.463219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.463198 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.495766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.495730 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 13:06:02.495766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.495757 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 13:06:02.495978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.495765 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7q7n9\"" Oct 30 13:06:02.498587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.498555 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925"] Oct 30 13:06:02.508446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.508419 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.508614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.508453 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.508614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.508481 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.588778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.588744 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v"] Oct 30 13:06:02.592498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.592472 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.609134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.609108 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.609259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.609144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.609301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.609278 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.609449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.609428 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.609532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.609460 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.610445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.610424 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6928j\"" Oct 30 13:06:02.612206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.612175 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v"] Oct 30 13:06:02.628827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.628801 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.710893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.710793 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g894\" (UniqueName: \"kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.711085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.710950 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.711085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.711008 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.778656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.778618 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:02.792490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.792461 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv"] Oct 30 13:06:02.795976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.795956 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.811394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.811368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.811534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.811407 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.811534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.811427 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g894\" (UniqueName: \"kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.811715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.811694 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.811790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.811769 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.818570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.818551 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 13:06:02.818759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.818741 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 13:06:02.819057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.819042 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 13:06:02.819376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.819360 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 13:06:02.819620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.819473 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 13:06:02.819620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.819493 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 13:06:02.819620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.819503 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fqpnj\"" Oct 30 13:06:02.839750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.836546 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv"] Oct 30 13:06:02.839750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.838612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g894\" (UniqueName: \"kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.902620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.902565 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.908568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.908541 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:02.912712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.912692 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.912814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.912720 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47twp\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.912814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.912746 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.912946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.912925 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.913007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.912990 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.913051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.913023 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:02.913086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:02.913069 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014296 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014342 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014365 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014381 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014407 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014427 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47twp\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.014576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.014452 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.015049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.015019 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.016932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.016909 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.017044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.016947 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.017154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.017136 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.017211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.017164 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.034306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.034280 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.036428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.036400 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47twp\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp\") pod \"istiod-openshift-gateway-7876bfcb74-pkvlv\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.075348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.075314 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:03.085500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.085473 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925"] Oct 30 13:06:03.089609 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:06:03.089568 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fbe9a2_92d6_4453_a8c0_95b2a7c726ae.slice/crio-c154609f267e0552aef7bb8d83aa0bd9b60f12f8ad955ae492d227e513dd2afe WatchSource:0}: Error finding container c154609f267e0552aef7bb8d83aa0bd9b60f12f8ad955ae492d227e513dd2afe: Status 404 returned error can't find the container with id c154609f267e0552aef7bb8d83aa0bd9b60f12f8ad955ae492d227e513dd2afe Oct 30 13:06:03.109418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.109395 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:03.201580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.201546 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v"] Oct 30 13:06:03.205684 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:06:03.205656 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3889d384_6946_4327_a67f_ce145b801baa.slice/crio-2f2d50806544358f5f4ef1ca4a7baa1a0388b65f1499e57c5dcc518ddb2bf9cc WatchSource:0}: Error finding container 2f2d50806544358f5f4ef1ca4a7baa1a0388b65f1499e57c5dcc518ddb2bf9cc: Status 404 returned error can't find the container with id 2f2d50806544358f5f4ef1ca4a7baa1a0388b65f1499e57c5dcc518ddb2bf9cc Oct 30 13:06:03.410520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.410426 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv"] Oct 30 13:06:03.413693 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:06:03.413664 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf56a52d_350e_4eed_b0ba_c1499bf00891.slice/crio-5b261a2675114841b85d6cbc6648bbc5e0a0eb64404dda7a318ea6a6c84aabae WatchSource:0}: Error finding container 5b261a2675114841b85d6cbc6648bbc5e0a0eb64404dda7a318ea6a6c84aabae: Status 404 returned error can't find the container with id 5b261a2675114841b85d6cbc6648bbc5e0a0eb64404dda7a318ea6a6c84aabae Oct 30 13:06:03.758496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.758444 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:03.827422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.827369 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" event={"ID":"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae","Type":"ContainerStarted","Data":"c154609f267e0552aef7bb8d83aa0bd9b60f12f8ad955ae492d227e513dd2afe"} Oct 30 13:06:03.828855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.828820 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" event={"ID":"af56a52d-350e-4eed-b0ba-c1499bf00891","Type":"ContainerStarted","Data":"5b261a2675114841b85d6cbc6648bbc5e0a0eb64404dda7a318ea6a6c84aabae"} Oct 30 13:06:03.830210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:03.830180 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" event={"ID":"3889d384-6946-4327-a67f-ce145b801baa","Type":"ContainerStarted","Data":"2f2d50806544358f5f4ef1ca4a7baa1a0388b65f1499e57c5dcc518ddb2bf9cc"} Oct 30 13:06:04.100905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:04.100817 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:06.634758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:06.634716 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:06.635096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:06.634801 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:06.850376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:06.850126 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" event={"ID":"af56a52d-350e-4eed-b0ba-c1499bf00891","Type":"ContainerStarted","Data":"a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959"} Oct 30 13:06:06.850376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:06.850284 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:06.917140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:06.917044 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" podStartSLOduration=1.698911132 podStartE2EDuration="4.917027159s" totalImagesPullingTime="3.218116027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:06.904799551 +0000 UTC m=+142.771417725" watchObservedRunningTime="2026-10-30 13:06:06.917027159 +0000 UTC m=+142.783645334" Oct 30 13:06:07.439275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:07.439239 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:07.644048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:07.644011 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:07.857847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:07.857809 5693 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-pkvlv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:06:07.858017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:07.857878 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" podUID="af56a52d-350e-4eed-b0ba-c1499bf00891" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:06:08.398803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:08.398769 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:08.555317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:08.555282 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:08.696034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:08.695957 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.856677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:10.856643 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:06:10.869315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:10.869284 5693 generic.go:356] "Generic (PLEG): container finished" podID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerID="55c3739c28c442104bed84c2563ca1db4e0a74f86d32ac3b323d162ee5acf399" exitCode=0 Oct 30 13:06:10.869464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:10.869367 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" event={"ID":"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae","Type":"ContainerDied","Data":"55c3739c28c442104bed84c2563ca1db4e0a74f86d32ac3b323d162ee5acf399"} Oct 30 13:06:10.870988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:10.870952 5693 generic.go:356] "Generic (PLEG): container finished" podID="3889d384-6946-4327-a67f-ce145b801baa" containerID="788a33e0f22c0c47b38a83cfe61cc12e37703fbac8d09321e6eadbf9c5b85aee" exitCode=0 Oct 30 13:06:10.870988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:10.870976 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" event={"ID":"3889d384-6946-4327-a67f-ce145b801baa","Type":"ContainerDied","Data":"788a33e0f22c0c47b38a83cfe61cc12e37703fbac8d09321e6eadbf9c5b85aee"} Oct 30 13:06:11.877546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:11.877456 5693 generic.go:356] "Generic (PLEG): container finished" podID="3889d384-6946-4327-a67f-ce145b801baa" containerID="a3fca771cb39736519d713d37ad9ea0f828d0d8be1cfa0628ee0b1bdb08bef02" exitCode=0 Oct 30 13:06:11.878033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:11.877538 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" event={"ID":"3889d384-6946-4327-a67f-ce145b801baa","Type":"ContainerDied","Data":"a3fca771cb39736519d713d37ad9ea0f828d0d8be1cfa0628ee0b1bdb08bef02"} Oct 30 13:06:12.139796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:12.139713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:13.887933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:13.887895 5693 generic.go:356] "Generic (PLEG): container finished" podID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerID="f4f922aea484d712c80d6417226bd98a2ecd9204c25ecf5f158d0153284877ef" exitCode=0 Oct 30 13:06:13.888390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:13.887988 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" event={"ID":"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae","Type":"ContainerDied","Data":"f4f922aea484d712c80d6417226bd98a2ecd9204c25ecf5f158d0153284877ef"} Oct 30 13:06:18.913673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:18.913634 5693 generic.go:356] "Generic (PLEG): container finished" podID="3889d384-6946-4327-a67f-ce145b801baa" containerID="e0ad3aef5273079846d8dffe268af17fb7b13624ca59dc656b9363096f3dc312" exitCode=0 Oct 30 13:06:18.914105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:18.913717 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" event={"ID":"3889d384-6946-4327-a67f-ce145b801baa","Type":"ContainerDied","Data":"e0ad3aef5273079846d8dffe268af17fb7b13624ca59dc656b9363096f3dc312"} Oct 30 13:06:20.056805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.056499 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:20.178765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.178684 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle\") pod \"3889d384-6946-4327-a67f-ce145b801baa\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " Oct 30 13:06:20.178765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.178741 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894\" (UniqueName: \"kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894\") pod \"3889d384-6946-4327-a67f-ce145b801baa\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " Oct 30 13:06:20.179036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.178782 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util\") pod \"3889d384-6946-4327-a67f-ce145b801baa\" (UID: \"3889d384-6946-4327-a67f-ce145b801baa\") " Oct 30 13:06:20.179313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.179286 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle" pod "3889d384-6946-4327-a67f-ce145b801baa" (UID: "3889d384-6946-4327-a67f-ce145b801baa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:20.180886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.180861 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894" pod "3889d384-6946-4327-a67f-ce145b801baa" (UID: "3889d384-6946-4327-a67f-ce145b801baa"). InnerVolumeSpecName "kube-api-access-8g894". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:20.182875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.182851 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util" pod "3889d384-6946-4327-a67f-ce145b801baa" (UID: "3889d384-6946-4327-a67f-ce145b801baa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:20.279953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.279928 5693 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-bundle\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:06:20.279953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.279954 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g894\" (UniqueName: \"kubernetes.io/projected/3889d384-6946-4327-a67f-ce145b801baa-kube-api-access-8g894\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:06:20.280098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.279964 5693 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3889d384-6946-4327-a67f-ce145b801baa-util\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:06:20.923156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.923117 5693 generic.go:356] "Generic (PLEG): container finished" podID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerID="6dcafc97e11c5962d447bbba4c98e14e488f34f3829ed2ea2968bcd889045e70" exitCode=0 Oct 30 13:06:20.923346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.923191 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" event={"ID":"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae","Type":"ContainerDied","Data":"6dcafc97e11c5962d447bbba4c98e14e488f34f3829ed2ea2968bcd889045e70"} Oct 30 13:06:20.925060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.925037 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" event={"ID":"3889d384-6946-4327-a67f-ce145b801baa","Type":"ContainerDied","Data":"2f2d50806544358f5f4ef1ca4a7baa1a0388b65f1499e57c5dcc518ddb2bf9cc"} Oct 30 13:06:20.925164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.925066 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mbk8v" Oct 30 13:06:20.925164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:20.925079 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2d50806544358f5f4ef1ca4a7baa1a0388b65f1499e57c5dcc518ddb2bf9cc" Oct 30 13:06:22.048162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.048138 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:22.196340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.196257 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle\") pod \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " Oct 30 13:06:22.196340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.196307 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j\") pod \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " Oct 30 13:06:22.196522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.196369 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util\") pod \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\" (UID: \"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae\") " Oct 30 13:06:22.196742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.196721 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle" pod "56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" (UID: "56fbe9a2-92d6-4453-a8c0-95b2a7c726ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:22.198516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.198487 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j" pod "56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" (UID: "56fbe9a2-92d6-4453-a8c0-95b2a7c726ae"). InnerVolumeSpecName "kube-api-access-b2h9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:22.201839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.201811 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util" pod "56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" (UID: "56fbe9a2-92d6-4453-a8c0-95b2a7c726ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:22.297255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.297225 5693 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-bundle\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.297255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.297249 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-kube-api-access-b2h9j\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.297255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.297258 5693 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fbe9a2-92d6-4453-a8c0-95b2a7c726ae-util\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.802123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.802090 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.933561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.933527 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" event={"ID":"56fbe9a2-92d6-4453-a8c0-95b2a7c726ae","Type":"ContainerDied","Data":"c154609f267e0552aef7bb8d83aa0bd9b60f12f8ad955ae492d227e513dd2afe"} Oct 30 13:06:22.933742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.933573 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c154609f267e0552aef7bb8d83aa0bd9b60f12f8ad955ae492d227e513dd2afe" Oct 30 13:06:22.933742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.933578 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ct925" Oct 30 13:06:22.942803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:22.942781 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.053948 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:06:23.053865 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:06:27.181215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:27.181174 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.317955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:27.317885 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:31.920553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:31.920513 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:36.506698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:36.506664 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:36.651821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:36.651790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.167432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.167392 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.324858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.324825 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd"] Oct 30 13:06:37.325154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325126 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="extract" Oct 30 13:06:37.325154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325142 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="extract" Oct 30 13:06:37.325154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325154 5693 cpu_manager.go:414] "containerMap: removing container" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="util" Oct 30 13:06:37.325154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325159 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="util" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325181 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="pull" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325187 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="pull" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325196 5693 cpu_manager.go:414] "containerMap: removing container" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="extract" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325205 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="extract" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325216 5693 cpu_manager.go:414] "containerMap: removing container" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="pull" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325220 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="pull" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325229 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="util" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325234 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="util" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325319 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fbe9a2-92d6-4453-a8c0-95b2a7c726ae" containerName="extract" Oct 30 13:06:37.325394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.325334 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3889d384-6946-4327-a67f-ce145b801baa" containerName="extract" Oct 30 13:06:37.327966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.327948 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.366545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.366517 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd"] Oct 30 13:06:37.397765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.397737 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 13:06:37.397877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.397792 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gfqwn\"" Oct 30 13:06:37.397877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.397815 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 13:06:37.397877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.397870 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 13:06:37.407953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.407928 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 13:06:37.408177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.408159 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 13:06:37.425455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.425393 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.425455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.425440 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.425657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.425478 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7549f34e-159f-499c-9599-7c9a4d55b06e-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.425657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.425556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.425766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.425699 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8tt\" (UniqueName: \"kubernetes.io/projected/7549f34e-159f-499c-9599-7c9a4d55b06e-kube-api-access-5m8tt\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.425766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.425757 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7549f34e-159f-499c-9599-7c9a4d55b06e-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.500072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.500041 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.526338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526308 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.526789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526348 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.526789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7549f34e-159f-499c-9599-7c9a4d55b06e-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.526789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526386 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.526789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526405 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8tt\" (UniqueName: \"kubernetes.io/projected/7549f34e-159f-499c-9599-7c9a4d55b06e-kube-api-access-5m8tt\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.526789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526450 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7549f34e-159f-499c-9599-7c9a4d55b06e-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.527000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.526795 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7549f34e-159f-499c-9599-7c9a4d55b06e-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.528962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.528925 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.529074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.529012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.529142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.529111 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7549f34e-159f-499c-9599-7c9a4d55b06e-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.529213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.529198 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7549f34e-159f-499c-9599-7c9a4d55b06e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.545878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.545854 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8tt\" (UniqueName: \"kubernetes.io/projected/7549f34e-159f-499c-9599-7c9a4d55b06e-kube-api-access-5m8tt\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd\" (UID: \"7549f34e-159f-499c-9599-7c9a4d55b06e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.641277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.641248 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:37.666271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.666242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.845326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.845286 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.996293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:37.996224 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd"] Oct 30 13:06:38.000334 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:06:38.000308 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7549f34e_159f_499c_9599_7c9a4d55b06e.slice/crio-d17c8ccfb7d0556bc1b5de0cd9035785d5358dca492515509686ff91dc77d9ad WatchSource:0}: Error finding container d17c8ccfb7d0556bc1b5de0cd9035785d5358dca492515509686ff91dc77d9ad: Status 404 returned error can't find the container with id d17c8ccfb7d0556bc1b5de0cd9035785d5358dca492515509686ff91dc77d9ad Oct 30 13:06:38.294530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:38.294458 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.491447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:38.491407 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.993812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:38.993771 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" event={"ID":"7549f34e-159f-499c-9599-7c9a4d55b06e","Type":"ContainerStarted","Data":"d17c8ccfb7d0556bc1b5de0cd9035785d5358dca492515509686ff91dc77d9ad"} Oct 30 13:06:40.510251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:40.510213 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:40.653097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:40.653030 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:41.003827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:41.003792 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" event={"ID":"7549f34e-159f-499c-9599-7c9a4d55b06e","Type":"ContainerStarted","Data":"fb1df3a4a59b1e8c0435c1a40ac7ac84f117cdc0335dc06abc7cec4760a82395"} Oct 30 13:06:41.747293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:41.747256 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:42.008818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:42.008730 5693 generic.go:356] "Generic (PLEG): container finished" podID="7549f34e-159f-499c-9599-7c9a4d55b06e" containerID="fb1df3a4a59b1e8c0435c1a40ac7ac84f117cdc0335dc06abc7cec4760a82395" exitCode=0 Oct 30 13:06:42.008963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:42.008816 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" event={"ID":"7549f34e-159f-499c-9599-7c9a4d55b06e","Type":"ContainerDied","Data":"fb1df3a4a59b1e8c0435c1a40ac7ac84f117cdc0335dc06abc7cec4760a82395"} Oct 30 13:06:43.014941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:43.014907 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" event={"ID":"7549f34e-159f-499c-9599-7c9a4d55b06e","Type":"ContainerStarted","Data":"5e971ec03e8e06e9595d6532777367e489ec784aa489475717bce7dac01baae9"} Oct 30 13:06:43.015384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:43.015062 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:43.107535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:43.107486 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" podStartSLOduration=2.364114528 podStartE2EDuration="6.107473195s" totalImagesPullingTime="3.199113654s" totalInitContainerRuntime="544.245013ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:43.095018659 +0000 UTC m=+178.961636855" watchObservedRunningTime="2026-10-30 13:06:43.107473195 +0000 UTC m=+178.974091365" Oct 30 13:06:45.122775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:45.122737 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:45.336837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:45.336805 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:48.509941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:48.509908 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:48.767418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:48.767325 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.454240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:50.454208 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.943167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:50.943133 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:51.118500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:51.118466 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:53.053620 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:06:53.053561 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:06:53.288878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:53.288841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:53.648567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:53.648534 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:54.021576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:54.021549 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-w6hnd" Oct 30 13:06:54.205869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:54.205830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:54.362415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:54.362334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:54.536645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:54.536589 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.053232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:57.053160 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.243220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:57.243184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:58.632691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:58.632649 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:59.945464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:06:59.945427 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:00.933764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:00.933731 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:01.081409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:01.081380 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.101939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:03.101905 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.261797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:03.261767 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:04.961616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:04.961559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.085566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:06.085532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.340525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:06.340441 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.699990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:06.699914 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.926145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:06.926112 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:07.340555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:07.340523 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:07.639668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:07.639572 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:12.296630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:12.296580 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:12.448263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:12.448229 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:12.690032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:12.689946 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:12.839963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:12.839931 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:13.943197 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:13.943165 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:14.129893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:14.129858 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:15.908613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:15.908575 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.055363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:16.055334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.208405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:16.208297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.369460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:16.369426 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.564182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:16.564145 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.778713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:16.778678 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.053980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.053946 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.145162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.144150 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7998d67549-9b9rf"] Oct 30 13:07:17.152676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.152647 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.175786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.175764 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 13:07:17.175923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.175865 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5nkmk\"" Oct 30 13:07:17.175923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.175865 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 13:07:17.189026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.188999 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7998d67549-9b9rf"] Oct 30 13:07:17.272274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.272235 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6757ec91-0fed-4966-a03d-ace6e941ed75-tmp\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.272449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.272320 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6757ec91-0fed-4966-a03d-ace6e941ed75-tls-certs\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.272449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.272404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9nt\" (UniqueName: \"kubernetes.io/projected/6757ec91-0fed-4966-a03d-ace6e941ed75-kube-api-access-2c9nt\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.287378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.287332 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.373545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.373466 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6757ec91-0fed-4966-a03d-ace6e941ed75-tmp\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.373735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.373636 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6757ec91-0fed-4966-a03d-ace6e941ed75-tls-certs\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.373735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.373672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9nt\" (UniqueName: \"kubernetes.io/projected/6757ec91-0fed-4966-a03d-ace6e941ed75-kube-api-access-2c9nt\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.376396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.376370 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6757ec91-0fed-4966-a03d-ace6e941ed75-tls-certs\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.376565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.376389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6757ec91-0fed-4966-a03d-ace6e941ed75-tmp\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.406024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.405998 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9nt\" (UniqueName: \"kubernetes.io/projected/6757ec91-0fed-4966-a03d-ace6e941ed75-kube-api-access-2c9nt\") pod \"kube-auth-proxy-7998d67549-9b9rf\" (UID: \"6757ec91-0fed-4966-a03d-ace6e941ed75\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.464613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.464562 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" Oct 30 13:07:17.482027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.481984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.678862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.678774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.765844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.765812 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7998d67549-9b9rf"] Oct 30 13:07:17.770023 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:07:17.769997 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6757ec91_0fed_4966_a03d_ace6e941ed75.slice/crio-cf0148fbb9e9650983bc4b33c97fd17b8f18f0a7481156501743397783128fe3 WatchSource:0}: Error finding container cf0148fbb9e9650983bc4b33c97fd17b8f18f0a7481156501743397783128fe3: Status 404 returned error can't find the container with id cf0148fbb9e9650983bc4b33c97fd17b8f18f0a7481156501743397783128fe3 Oct 30 13:07:17.882711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:17.882669 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.157387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.157347 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" event={"ID":"6757ec91-0fed-4966-a03d-ace6e941ed75","Type":"ContainerStarted","Data":"cf0148fbb9e9650983bc4b33c97fd17b8f18f0a7481156501743397783128fe3"} Oct 30 13:07:18.486756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.486725 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.669699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.669665 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:18.673371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.673342 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.709078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.709048 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 13:07:18.709225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.709167 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 13:07:18.720876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.720821 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 13:07:18.720876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.720826 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x4w5v\"" Oct 30 13:07:18.722364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.722342 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:18.740302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.740215 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.790205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.790174 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file\") pod \"limitador-limitador-74dfcfb894-f5mgd\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.790359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.790300 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vq4n\" (UniqueName: \"kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n\") pod \"limitador-limitador-74dfcfb894-f5mgd\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.845641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.845579 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:18.890840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.890804 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file\") pod \"limitador-limitador-74dfcfb894-f5mgd\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.890991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.890870 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vq4n\" (UniqueName: \"kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n\") pod \"limitador-limitador-74dfcfb894-f5mgd\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.891548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.891526 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file\") pod \"limitador-limitador-74dfcfb894-f5mgd\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.920266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.920233 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vq4n\" (UniqueName: \"kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n\") pod \"limitador-limitador-74dfcfb894-f5mgd\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:18.963521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.963481 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.990634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:18.990533 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:19.217944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:19.217904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:19.311343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:19.310420 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:19.315264 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:07:19.315227 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e2a19a_a4c2_4aa7_a58a_90575a524086.slice/crio-2e23306eaf6d10bdf6329f904366c846737741bec53b61f67a9e9494df78224e WatchSource:0}: Error finding container 2e23306eaf6d10bdf6329f904366c846737741bec53b61f67a9e9494df78224e: Status 404 returned error can't find the container with id 2e23306eaf6d10bdf6329f904366c846737741bec53b61f67a9e9494df78224e Oct 30 13:07:19.869613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:19.869568 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:20.061718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:20.061681 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:20.169156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:20.169084 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" event={"ID":"f6e2a19a-a4c2-4aa7-a58a-90575a524086","Type":"ContainerStarted","Data":"2e23306eaf6d10bdf6329f904366c846737741bec53b61f67a9e9494df78224e"} Oct 30 13:07:21.175435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:21.175394 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" event={"ID":"6757ec91-0fed-4966-a03d-ace6e941ed75","Type":"ContainerStarted","Data":"c05c2cdab20819c1b4e2582d9098621c0463a7ac38a3dccd1a9e929880462005"} Oct 30 13:07:22.181082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:22.181043 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" event={"ID":"f6e2a19a-a4c2-4aa7-a58a-90575a524086","Type":"ContainerStarted","Data":"0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f"} Oct 30 13:07:22.181557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:22.181267 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:22.221058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:22.220993 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7998d67549-9b9rf" podStartSLOduration=2.7300317769999998 podStartE2EDuration="5.220974837s" totalImagesPullingTime="2.49094306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:21.223020241 +0000 UTC m=+217.089638432" watchObservedRunningTime="2026-10-30 13:07:22.220974837 +0000 UTC m=+218.087593089" Oct 30 13:07:22.221725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:22.221689 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" podStartSLOduration=1.9679878720000001 podStartE2EDuration="4.221680189s" totalImagesPullingTime="2.253692317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:22.220606701 +0000 UTC m=+218.087224867" watchObservedRunningTime="2026-10-30 13:07:22.221680189 +0000 UTC m=+218.088298372" Oct 30 13:07:23.054313 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:07:23.054271 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:07:23.974670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:23.974637 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.968419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:24.968381 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9"] Oct 30 13:07:24.971720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:24.971699 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:24.986174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:24.986148 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 13:07:24.997931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:24.997877 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 13:07:24.998085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:24.997983 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-9w8vj\"" Oct 30 13:07:25.022726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.022695 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 13:07:25.023529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.023509 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9"] Oct 30 13:07:25.149072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.149031 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9jv\" (UniqueName: \"kubernetes.io/projected/034a48e4-ba7c-4a2a-ae39-e64831996044-kube-api-access-fn9jv\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.149241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.149168 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/034a48e4-ba7c-4a2a-ae39-e64831996044-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.149241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.149201 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/034a48e4-ba7c-4a2a-ae39-e64831996044-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.149241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.149224 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/034a48e4-ba7c-4a2a-ae39-e64831996044-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.250347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.250305 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/034a48e4-ba7c-4a2a-ae39-e64831996044-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.250542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.250402 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/034a48e4-ba7c-4a2a-ae39-e64831996044-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.250542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.250427 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/034a48e4-ba7c-4a2a-ae39-e64831996044-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.250542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.250487 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9jv\" (UniqueName: \"kubernetes.io/projected/034a48e4-ba7c-4a2a-ae39-e64831996044-kube-api-access-fn9jv\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.250776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.250754 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/034a48e4-ba7c-4a2a-ae39-e64831996044-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.250776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.250766 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/034a48e4-ba7c-4a2a-ae39-e64831996044-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.251382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.251357 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/034a48e4-ba7c-4a2a-ae39-e64831996044-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.270591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.270559 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9jv\" (UniqueName: \"kubernetes.io/projected/034a48e4-ba7c-4a2a-ae39-e64831996044-kube-api-access-fn9jv\") pod \"kserve-module-controller-manager-7566cf5fc8-9xtc9\" (UID: \"034a48e4-ba7c-4a2a-ae39-e64831996044\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.284399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.284380 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:25.590481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:25.590403 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9"] Oct 30 13:07:25.594981 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:07:25.594947 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034a48e4_ba7c_4a2a_ae39_e64831996044.slice/crio-770b3e82ff381b0da26e59db9b1fe7cd786d8f4253a0fda688873f2c1a086de9 WatchSource:0}: Error finding container 770b3e82ff381b0da26e59db9b1fe7cd786d8f4253a0fda688873f2c1a086de9: Status 404 returned error can't find the container with id 770b3e82ff381b0da26e59db9b1fe7cd786d8f4253a0fda688873f2c1a086de9 Oct 30 13:07:26.011777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:26.011741 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.199072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:26.199034 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" event={"ID":"034a48e4-ba7c-4a2a-ae39-e64831996044","Type":"ContainerStarted","Data":"770b3e82ff381b0da26e59db9b1fe7cd786d8f4253a0fda688873f2c1a086de9"} Oct 30 13:07:28.207842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:28.207808 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" event={"ID":"034a48e4-ba7c-4a2a-ae39-e64831996044","Type":"ContainerStarted","Data":"b5056453b33f15994a1b8d590b9abb2f31e1754220ae078439eb25ab0496d175"} Oct 30 13:07:28.208229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:28.207941 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:28.263798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:28.263747 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" podStartSLOduration=2.6116767100000002 podStartE2EDuration="4.263735518s" totalImagesPullingTime="1.652058808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:28.263366003 +0000 UTC m=+224.129984177" watchObservedRunningTime="2026-10-30 13:07:28.263735518 +0000 UTC m=+224.130353707" Oct 30 13:07:31.945795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:31.945760 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:32.184156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:32.184099 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:32.917318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:32.917159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:33.135073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:33.135040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:33.186353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:33.186274 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:35.051450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.051412 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:35.051961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.051736 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" podUID="f6e2a19a-a4c2-4aa7-a58a-90575a524086" containerName="limitador" containerID="cri-o://0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f" gracePeriod=30 Oct 30 13:07:35.585779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.585752 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:35.636806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.636726 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n\" (UniqueName: \"kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n\") pod \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " Oct 30 13:07:35.636962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.636842 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file\") pod \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\" (UID: \"f6e2a19a-a4c2-4aa7-a58a-90575a524086\") " Oct 30 13:07:35.637203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.637178 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file" pod "f6e2a19a-a4c2-4aa7-a58a-90575a524086" (UID: "f6e2a19a-a4c2-4aa7-a58a-90575a524086"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:35.638967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.638936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n" pod "f6e2a19a-a4c2-4aa7-a58a-90575a524086" (UID: "f6e2a19a-a4c2-4aa7-a58a-90575a524086"). InnerVolumeSpecName "kube-api-access-6vq4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:35.706620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.706547 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:35.737841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.737811 5693 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2a19a-a4c2-4aa7-a58a-90575a524086-config-file\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:07:35.737841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.737841 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vq4n\" (UniqueName: \"kubernetes.io/projected/f6e2a19a-a4c2-4aa7-a58a-90575a524086-kube-api-access-6vq4n\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:07:35.860581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:35.860544 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:36.240421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.240384 5693 generic.go:356] "Generic (PLEG): container finished" podID="f6e2a19a-a4c2-4aa7-a58a-90575a524086" containerID="0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f" exitCode=0 Oct 30 13:07:36.240921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.240434 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" Oct 30 13:07:36.240921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.240447 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" event={"ID":"f6e2a19a-a4c2-4aa7-a58a-90575a524086","Type":"ContainerDied","Data":"0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f"} Oct 30 13:07:36.240921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.240488 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f5mgd" event={"ID":"f6e2a19a-a4c2-4aa7-a58a-90575a524086","Type":"ContainerDied","Data":"2e23306eaf6d10bdf6329f904366c846737741bec53b61f67a9e9494df78224e"} Oct 30 13:07:36.240921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.240510 5693 scope.go:122] "RemoveContainer" containerID="0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f" Oct 30 13:07:36.250706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.250679 5693 scope.go:122] "RemoveContainer" containerID="0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f" Oct 30 13:07:36.251252 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:07:36.251226 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f\": container with ID starting with 0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f not found: ID does not exist" containerID="0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f" Oct 30 13:07:36.251345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.251264 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f"} err="failed to get container status \"0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f\": rpc error: code = NotFound desc = could not find container \"0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f\": container with ID starting with 0b40ad1b380052706ea7bcbe968d21df52ae8ddfb75d843a06a901d97287c97f not found: ID does not exist" Oct 30 13:07:36.290399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.290364 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:36.302382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:36.302357 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f5mgd"] Oct 30 13:07:38.251445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:38.251403 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e2a19a-a4c2-4aa7-a58a-90575a524086" path="/var/lib/kubelet/pods/f6e2a19a-a4c2-4aa7-a58a-90575a524086/volumes" Oct 30 13:07:38.710615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:38.710528 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:38.866904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:38.866860 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.160277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.160243 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.352502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.352467 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.818345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.818305 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf"] Oct 30 13:07:39.818872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.818846 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f6e2a19a-a4c2-4aa7-a58a-90575a524086" containerName="limitador" Oct 30 13:07:39.819055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.819027 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e2a19a-a4c2-4aa7-a58a-90575a524086" containerName="limitador" Oct 30 13:07:39.819189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.819144 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e2a19a-a4c2-4aa7-a58a-90575a524086" containerName="limitador" Oct 30 13:07:39.823538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.823521 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:39.839491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.839470 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 13:07:39.840243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.840221 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf"] Oct 30 13:07:39.851053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.851031 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:39.851156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.851078 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:39.872018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.871992 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dnd\" (UniqueName: \"kubernetes.io/projected/9b500b45-7eed-428d-9bbd-28bc8ac1576d-kube-api-access-r2dnd\") pod \"postgres-7cdbf6dc44-vmntf\" (UID: \"9b500b45-7eed-428d-9bbd-28bc8ac1576d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:39.872152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.872111 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b500b45-7eed-428d-9bbd-28bc8ac1576d-data\") pod \"postgres-7cdbf6dc44-vmntf\" (UID: \"9b500b45-7eed-428d-9bbd-28bc8ac1576d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:39.939361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.939327 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wt246\"" Oct 30 13:07:39.957618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.957570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.972817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.972790 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b500b45-7eed-428d-9bbd-28bc8ac1576d-data\") pod \"postgres-7cdbf6dc44-vmntf\" (UID: \"9b500b45-7eed-428d-9bbd-28bc8ac1576d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:39.972922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.972843 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dnd\" (UniqueName: \"kubernetes.io/projected/9b500b45-7eed-428d-9bbd-28bc8ac1576d-kube-api-access-r2dnd\") pod \"postgres-7cdbf6dc44-vmntf\" (UID: \"9b500b45-7eed-428d-9bbd-28bc8ac1576d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:39.973162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.973144 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b500b45-7eed-428d-9bbd-28bc8ac1576d-data\") pod \"postgres-7cdbf6dc44-vmntf\" (UID: \"9b500b45-7eed-428d-9bbd-28bc8ac1576d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:39.998265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:39.998236 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dnd\" (UniqueName: \"kubernetes.io/projected/9b500b45-7eed-428d-9bbd-28bc8ac1576d-kube-api-access-r2dnd\") pod \"postgres-7cdbf6dc44-vmntf\" (UID: \"9b500b45-7eed-428d-9bbd-28bc8ac1576d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:40.136471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:40.136355 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:40.429847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:40.429774 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf"] Oct 30 13:07:40.434217 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:07:40.434183 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b500b45_7eed_428d_9bbd_28bc8ac1576d.slice/crio-873845c9b9b3cb52b42375160974676351faa48067224c064e9566642a9fc042 WatchSource:0}: Error finding container 873845c9b9b3cb52b42375160974676351faa48067224c064e9566642a9fc042: Status 404 returned error can't find the container with id 873845c9b9b3cb52b42375160974676351faa48067224c064e9566642a9fc042 Oct 30 13:07:41.263607 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:41.263566 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" event={"ID":"9b500b45-7eed-428d-9bbd-28bc8ac1576d","Type":"ContainerStarted","Data":"873845c9b9b3cb52b42375160974676351faa48067224c064e9566642a9fc042"} Oct 30 13:07:44.617963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:44.617885 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:44.795172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:44.795138 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:46.864103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:46.864067 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 13:07:47.291953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:47.291919 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" event={"ID":"9b500b45-7eed-428d-9bbd-28bc8ac1576d","Type":"ContainerStarted","Data":"c0c1bebfdeb878b852c952fc52a155767fa2206cc26483314fd6c4ffe4b4b3f3"} Oct 30 13:07:47.292115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:47.291996 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:07:47.330393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:47.330328 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" podStartSLOduration=1.91976123 podStartE2EDuration="8.330310671s" totalImagesPullingTime="6.410549441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:47.329535817 +0000 UTC m=+243.196153991" watchObservedRunningTime="2026-10-30 13:07:47.330310671 +0000 UTC m=+243.196928848" Oct 30 13:07:49.212758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:49.212725 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9xtc9" Oct 30 13:07:51.205733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:51.205700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:53.053738 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:07:53.053703 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:07:53.324589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:07:53.324525 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vmntf" Oct 30 13:08:00.236023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.235980 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.286002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.285956 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74dcbcc48d-bhbvg"] Oct 30 13:08:00.292343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.292319 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.309310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.308970 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74dcbcc48d-bhbvg"] Oct 30 13:08:00.319730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.319698 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 13:08:00.319730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.319702 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:08:00.319919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.319745 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 13:08:00.319919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.319785 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7nw2c\"" Oct 30 13:08:00.351531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.351492 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.351736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.351588 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.351736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.351634 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdxg\" (UniqueName: \"kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.377485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.377454 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.452954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.452912 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.453117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.452970 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.453117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.452996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdxg\" (UniqueName: \"kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.455546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.455517 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.455821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.455800 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.472245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.472214 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdxg\" (UniqueName: \"kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg\") pod \"maas-controller-74dcbcc48d-bhbvg\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.515538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.515467 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.606096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.606065 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:00.653437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.653399 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.789894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.789801 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78c7bfc749-57x8m"] Oct 30 13:08:00.794390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.794368 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.814453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.814427 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c7bfc749-57x8m"] Oct 30 13:08:00.856512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.856474 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrv7\" (UniqueName: \"kubernetes.io/projected/53c26842-64be-41e0-b367-57e23106974b-kube-api-access-ttrv7\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.856690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.856520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c26842-64be-41e0-b367-57e23106974b-metrics-certs\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.856690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.856663 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53c26842-64be-41e0-b367-57e23106974b-webhook-cert\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.902440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.902409 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74dcbcc48d-bhbvg"] Oct 30 13:08:00.906902 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:08:00.906876 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe6efa9_c76b_478f_8ec8_dc7a6d045277.slice/crio-7d0382e1e28d95f0b11f72a327b20ddbf1979bb77a81853af0381be86e5451c6 WatchSource:0}: Error finding container 7d0382e1e28d95f0b11f72a327b20ddbf1979bb77a81853af0381be86e5451c6: Status 404 returned error can't find the container with id 7d0382e1e28d95f0b11f72a327b20ddbf1979bb77a81853af0381be86e5451c6 Oct 30 13:08:00.957367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.957333 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrv7\" (UniqueName: \"kubernetes.io/projected/53c26842-64be-41e0-b367-57e23106974b-kube-api-access-ttrv7\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.957561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.957385 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c26842-64be-41e0-b367-57e23106974b-metrics-certs\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.957561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.957415 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53c26842-64be-41e0-b367-57e23106974b-webhook-cert\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.959862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.959838 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c26842-64be-41e0-b367-57e23106974b-metrics-certs\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.959960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.959872 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53c26842-64be-41e0-b367-57e23106974b-webhook-cert\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:00.976310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:00.976285 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrv7\" (UniqueName: \"kubernetes.io/projected/53c26842-64be-41e0-b367-57e23106974b-kube-api-access-ttrv7\") pod \"maas-controller-78c7bfc749-57x8m\" (UID: \"53c26842-64be-41e0-b367-57e23106974b\") " pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:01.108149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:01.108063 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:01.352151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:01.352113 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" event={"ID":"cbe6efa9-c76b-478f-8ec8-dc7a6d045277","Type":"ContainerStarted","Data":"7d0382e1e28d95f0b11f72a327b20ddbf1979bb77a81853af0381be86e5451c6"} Oct 30 13:08:01.406358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:01.406284 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c7bfc749-57x8m"] Oct 30 13:08:01.410302 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:08:01.410274 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c26842_64be_41e0_b367_57e23106974b.slice/crio-f89ade01e648c73969f014a08b2ae0e4586e2dbb906c0648740fc5e0914db04c WatchSource:0}: Error finding container f89ade01e648c73969f014a08b2ae0e4586e2dbb906c0648740fc5e0914db04c: Status 404 returned error can't find the container with id f89ade01e648c73969f014a08b2ae0e4586e2dbb906c0648740fc5e0914db04c Oct 30 13:08:02.358827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:02.358729 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c7bfc749-57x8m" event={"ID":"53c26842-64be-41e0-b367-57e23106974b","Type":"ContainerStarted","Data":"f89ade01e648c73969f014a08b2ae0e4586e2dbb906c0648740fc5e0914db04c"} Oct 30 13:08:03.185550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:03.185516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:03.331002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:03.330970 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:04.116693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.116564 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:04.260933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.260904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:04.369621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.369508 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" event={"ID":"cbe6efa9-c76b-478f-8ec8-dc7a6d045277","Type":"ContainerStarted","Data":"909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602"} Oct 30 13:08:04.369789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.369717 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:04.371229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.371199 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c7bfc749-57x8m" event={"ID":"53c26842-64be-41e0-b367-57e23106974b","Type":"ContainerStarted","Data":"1fdc29a6f2448292944233825bf9293101ab743c2f7f7027274e3ed3d3df994d"} Oct 30 13:08:04.371358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.371319 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:04.410547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.410409 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" podStartSLOduration=1.488875618 podStartE2EDuration="4.410391795s" totalImagesPullingTime="2.921516177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:04.410194803 +0000 UTC m=+260.276812977" watchObservedRunningTime="2026-10-30 13:08:04.410391795 +0000 UTC m=+260.277009969" Oct 30 13:08:04.451441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:04.451382 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78c7bfc749-57x8m" podStartSLOduration=1.643053029 podStartE2EDuration="4.451362569s" totalImagesPullingTime="2.80830954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:04.450628463 +0000 UTC m=+260.317246630" watchObservedRunningTime="2026-10-30 13:08:04.451362569 +0000 UTC m=+260.317980829" Oct 30 13:08:05.565576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.565539 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s"] Oct 30 13:08:05.569353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.569324 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.589574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.589514 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 13:08:05.601756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.601733 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-pw74r\"" Oct 30 13:08:05.605456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.605433 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 13:08:05.605576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.605491 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 13:08:05.606963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.606897 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s"] Oct 30 13:08:05.707758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.707725 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqwh\" (UniqueName: \"kubernetes.io/projected/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-kube-api-access-2wqwh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.707758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.707759 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.707962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.707787 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.707962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.707928 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.741460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.741424 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:05.809332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.809290 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqwh\" (UniqueName: \"kubernetes.io/projected/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-kube-api-access-2wqwh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.809534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.809341 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.809534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.809381 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.809534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.809455 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.810286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.810256 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.810392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.810337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.811876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.811855 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.828831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.828804 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqwh\" (UniqueName: \"kubernetes.io/projected/de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1-kube-api-access-2wqwh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fhq4s\" (UID: \"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:05.876802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.876713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:05.882653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:05.882628 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:06.184540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:06.184431 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s"] Oct 30 13:08:06.188344 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:08:06.188317 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde229b37_17d3_4fbf_b0b9_5ffb01ff5cd1.slice/crio-4614e5c4a2428d37fe4263699d4a3cdd4fd188595eb4d920be1c5ab883d88201 WatchSource:0}: Error finding container 4614e5c4a2428d37fe4263699d4a3cdd4fd188595eb4d920be1c5ab883d88201: Status 404 returned error can't find the container with id 4614e5c4a2428d37fe4263699d4a3cdd4fd188595eb4d920be1c5ab883d88201 Oct 30 13:08:06.380375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:06.380340 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" event={"ID":"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1","Type":"ContainerStarted","Data":"4614e5c4a2428d37fe4263699d4a3cdd4fd188595eb4d920be1c5ab883d88201"} Oct 30 13:08:15.377347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.377305 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:15.378337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.378319 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:08:15.492032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.491996 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74dcbcc48d-bhbvg"] Oct 30 13:08:15.492227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.492187 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" podUID="cbe6efa9-c76b-478f-8ec8-dc7a6d045277" containerName="manager" containerID="cri-o://909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602" gracePeriod=30 Oct 30 13:08:15.760734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.760710 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:15.915397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.915338 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert\") pod \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " Oct 30 13:08:15.916270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.916238 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs\") pod \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " Oct 30 13:08:15.916388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.916345 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg\" (UniqueName: \"kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg\") pod \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\" (UID: \"cbe6efa9-c76b-478f-8ec8-dc7a6d045277\") " Oct 30 13:08:15.919208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.919182 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert" pod "cbe6efa9-c76b-478f-8ec8-dc7a6d045277" (UID: "cbe6efa9-c76b-478f-8ec8-dc7a6d045277"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:15.919594 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.919567 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg" pod "cbe6efa9-c76b-478f-8ec8-dc7a6d045277" (UID: "cbe6efa9-c76b-478f-8ec8-dc7a6d045277"). InnerVolumeSpecName "kube-api-access-cgdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:15.920797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:15.920771 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs" pod "cbe6efa9-c76b-478f-8ec8-dc7a6d045277" (UID: "cbe6efa9-c76b-478f-8ec8-dc7a6d045277"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:16.017587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.017546 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.017587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.017581 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdxg\" (UniqueName: \"kubernetes.io/projected/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-kube-api-access-cgdxg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.017587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.017592 5693 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbe6efa9-c76b-478f-8ec8-dc7a6d045277-webhook-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.429701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.429654 5693 generic.go:356] "Generic (PLEG): container finished" podID="cbe6efa9-c76b-478f-8ec8-dc7a6d045277" containerID="909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602" exitCode=0 Oct 30 13:08:16.430144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.429729 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" Oct 30 13:08:16.430144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.429732 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" event={"ID":"cbe6efa9-c76b-478f-8ec8-dc7a6d045277","Type":"ContainerDied","Data":"909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602"} Oct 30 13:08:16.430144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.429819 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dcbcc48d-bhbvg" event={"ID":"cbe6efa9-c76b-478f-8ec8-dc7a6d045277","Type":"ContainerDied","Data":"7d0382e1e28d95f0b11f72a327b20ddbf1979bb77a81853af0381be86e5451c6"} Oct 30 13:08:16.430144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.429849 5693 scope.go:122] "RemoveContainer" containerID="909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602" Oct 30 13:08:16.439631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.439592 5693 scope.go:122] "RemoveContainer" containerID="909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602" Oct 30 13:08:16.439909 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:08:16.439888 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602\": container with ID starting with 909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602 not found: ID does not exist" containerID="909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602" Oct 30 13:08:16.440002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.439917 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602"} err="failed to get container status \"909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602\": rpc error: code = NotFound desc = could not find container \"909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602\": container with ID starting with 909b8c91e1aca5eaa1b8cb02eb7d27b950e7267d9b49de345dbafc2d98028602 not found: ID does not exist" Oct 30 13:08:16.480522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.480494 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74dcbcc48d-bhbvg"] Oct 30 13:08:16.492235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:16.492212 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74dcbcc48d-bhbvg"] Oct 30 13:08:18.253328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:18.253297 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe6efa9-c76b-478f-8ec8-dc7a6d045277" path="/var/lib/kubelet/pods/cbe6efa9-c76b-478f-8ec8-dc7a6d045277/volumes" Oct 30 13:08:18.447417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:18.447359 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" event={"ID":"de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1","Type":"ContainerStarted","Data":"2a97ac984c5d47db70fe07b3448487b5bc5ef8705f2f6db04b739cdf85dfafbb"} Oct 30 13:08:18.447632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:18.447528 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:18.516552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:18.516454 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" podStartSLOduration=2.140712857 podStartE2EDuration="13.516441625s" totalImagesPullingTime="11.375728768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:18.515055134 +0000 UTC m=+274.381673306" watchObservedRunningTime="2026-10-30 13:08:18.516441625 +0000 UTC m=+274.383059798" Oct 30 13:08:21.686965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.686926 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d"] Oct 30 13:08:21.687515 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.687493 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cbe6efa9-c76b-478f-8ec8-dc7a6d045277" containerName="manager" Oct 30 13:08:21.687633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.687516 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe6efa9-c76b-478f-8ec8-dc7a6d045277" containerName="manager" Oct 30 13:08:21.687701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.687633 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe6efa9-c76b-478f-8ec8-dc7a6d045277" containerName="manager" Oct 30 13:08:21.694062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.694038 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.725117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.725090 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d"] Oct 30 13:08:21.771865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.771822 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmc49\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.772072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.771934 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.772072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.772048 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.772233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.772205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.772276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.772250 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.772276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.772272 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.772355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.772291 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.873609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873577 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.873784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873634 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.873784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873753 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.873904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873799 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.873904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873831 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.874006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873918 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmc49\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.874006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.873950 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.874677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.874651 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.876115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.876088 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.876238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.876169 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.876302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.876262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.876408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.876389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.892753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.892727 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:21.892895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:21.892876 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmc49\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tr48d\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:22.005277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.005248 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:22.319524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.319399 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d"] Oct 30 13:08:22.322646 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:08:22.322622 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64fe897_44aa_40ff_a273_5c40d458c4a2.slice/crio-55f7b5f245d8b6c29ecd5577c58c74713c1be42991b8a431685577aac8223cce WatchSource:0}: Error finding container 55f7b5f245d8b6c29ecd5577c58c74713c1be42991b8a431685577aac8223cce: Status 404 returned error can't find the container with id 55f7b5f245d8b6c29ecd5577c58c74713c1be42991b8a431685577aac8223cce Oct 30 13:08:22.324814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.324774 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:22.324898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.324837 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:22.466051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.466016 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" event={"ID":"a64fe897-44aa-40ff-a273-5c40d458c4a2","Type":"ContainerStarted","Data":"46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26"} Oct 30 13:08:22.466214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.466071 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" event={"ID":"a64fe897-44aa-40ff-a273-5c40d458c4a2","Type":"ContainerStarted","Data":"55f7b5f245d8b6c29ecd5577c58c74713c1be42991b8a431685577aac8223cce"} Oct 30 13:08:22.466214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.466099 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:22.511995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.511942 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" podStartSLOduration=1.511927005 podStartE2EDuration="1.511927005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:22.511442229 +0000 UTC m=+278.378060403" watchObservedRunningTime="2026-10-30 13:08:22.511927005 +0000 UTC m=+278.378545177" Oct 30 13:08:22.785085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.785040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:22.952097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:22.952015 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:23.056241 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:08:23.056166 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:08:23.106907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:23.106855 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:23.384075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:23.384000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:23.472344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:23.472313 5693 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tr48d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:08:23.472504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:23.472359 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" podUID="a64fe897-44aa-40ff-a273-5c40d458c4a2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:08:23.656724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:23.656645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.571043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:25.570964 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.705966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:25.705933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.845019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:25.844936 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.982712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:25.982669 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:26.471845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.471815 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 13:08:26.557530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.557501 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv"] Oct 30 13:08:26.557860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.557828 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" podUID="af56a52d-350e-4eed-b0ba-c1499bf00891" containerName="discovery" containerID="cri-o://a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959" gracePeriod=30 Oct 30 13:08:26.805159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.805135 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:08:26.919697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919665 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.919890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919755 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.919890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919785 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.919890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919839 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.920052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919893 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.920052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.920052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.919967 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token\") pod \"af56a52d-350e-4eed-b0ba-c1499bf00891\" (UID: \"af56a52d-350e-4eed-b0ba-c1499bf00891\") " Oct 30 13:08:26.920294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.920262 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:26.922141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.922086 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:26.922255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.922205 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:26.922318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.922259 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:26.922438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.922416 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:26.922507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.922436 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "kube-api-access-47twp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:26.922543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:26.922512 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token" pod "af56a52d-350e-4eed-b0ba-c1499bf00891" (UID: "af56a52d-350e-4eed-b0ba-c1499bf00891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021148 5693 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-cacerts\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021179 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47twp\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-kube-api-access-47twp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021189 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021199 5693 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-kubeconfig\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021209 5693 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56a52d-350e-4eed-b0ba-c1499bf00891-local-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021217 5693 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-dns-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.021225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.021225 5693 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af56a52d-350e-4eed-b0ba-c1499bf00891-istio-csr-ca-configmap\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:27.486852 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.486822 5693 generic.go:356] "Generic (PLEG): container finished" podID="af56a52d-350e-4eed-b0ba-c1499bf00891" containerID="a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959" exitCode=0 Oct 30 13:08:27.487017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.486885 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" event={"ID":"af56a52d-350e-4eed-b0ba-c1499bf00891","Type":"ContainerDied","Data":"a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959"} Oct 30 13:08:27.487017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.486924 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" event={"ID":"af56a52d-350e-4eed-b0ba-c1499bf00891","Type":"ContainerDied","Data":"5b261a2675114841b85d6cbc6648bbc5e0a0eb64404dda7a318ea6a6c84aabae"} Oct 30 13:08:27.487017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.486943 5693 scope.go:122] "RemoveContainer" containerID="a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959" Oct 30 13:08:27.487017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.486887 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv" Oct 30 13:08:27.495616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.495578 5693 scope.go:122] "RemoveContainer" containerID="a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959" Oct 30 13:08:27.495877 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:08:27.495857 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959\": container with ID starting with a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959 not found: ID does not exist" containerID="a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959" Oct 30 13:08:27.495924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.495884 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959"} err="failed to get container status \"a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959\": rpc error: code = NotFound desc = could not find container \"a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959\": container with ID starting with a121b86f7816a186cefc246ce4aade1ce3d2df24955ecd051db7583ab44ad959 not found: ID does not exist" Oct 30 13:08:27.546787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.546760 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv"] Oct 30 13:08:27.558689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:27.558664 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-pkvlv"] Oct 30 13:08:28.252648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:28.252615 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af56a52d-350e-4eed-b0ba-c1499bf00891" path="/var/lib/kubelet/pods/af56a52d-350e-4eed-b0ba-c1499bf00891/volumes" Oct 30 13:08:29.457579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:29.457543 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fhq4s" Oct 30 13:08:34.174105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:34.174070 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:34.375913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:34.375875 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:35.671466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:35.671434 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:35.810476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:35.810443 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:35.952025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:35.951936 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:36.198478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.198444 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7666d54c64-x2std"] Oct 30 13:08:36.198973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.198926 5693 cpu_manager.go:414] "containerMap: removing container" podUID="af56a52d-350e-4eed-b0ba-c1499bf00891" containerName="discovery" Oct 30 13:08:36.199120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.199011 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af56a52d-350e-4eed-b0ba-c1499bf00891" containerName="discovery" Oct 30 13:08:36.199248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.199169 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="af56a52d-350e-4eed-b0ba-c1499bf00891" containerName="discovery" Oct 30 13:08:36.205222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.205166 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.221827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.221800 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7666d54c64-x2std"] Oct 30 13:08:36.232429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.232405 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:08:36.237750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.237727 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gzx7f\"" Oct 30 13:08:36.237855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.237732 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 13:08:36.303176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.303139 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5gb\" (UniqueName: \"kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.303176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.303181 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.303370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.303209 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.303370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.303315 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.405814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.405764 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5gb\" (UniqueName: \"kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.406005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.405824 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.406005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.405877 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.406005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.405941 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.406803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.406776 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.408493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.408468 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.408493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.408481 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.425666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.425628 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5gb\" (UniqueName: \"kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb\") pod \"maas-api-7666d54c64-x2std\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.518199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.518161 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:36.818648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:36.818551 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7666d54c64-x2std"] Oct 30 13:08:36.822946 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:08:36.822920 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba780c5b_cbd2_48fd_b7d1_3679676a8e56.slice/crio-6b4047a3113ecf30991f838bead0eea58f51cb4ac1de8cd145a2e69a935e2bf2 WatchSource:0}: Error finding container 6b4047a3113ecf30991f838bead0eea58f51cb4ac1de8cd145a2e69a935e2bf2: Status 404 returned error can't find the container with id 6b4047a3113ecf30991f838bead0eea58f51cb4ac1de8cd145a2e69a935e2bf2 Oct 30 13:08:37.532404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:37.532361 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" event={"ID":"ba780c5b-cbd2-48fd-b7d1-3679676a8e56","Type":"ContainerStarted","Data":"6b4047a3113ecf30991f838bead0eea58f51cb4ac1de8cd145a2e69a935e2bf2"} Oct 30 13:08:38.537767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:38.537734 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" event={"ID":"ba780c5b-cbd2-48fd-b7d1-3679676a8e56","Type":"ContainerStarted","Data":"188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8"} Oct 30 13:08:38.538150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:38.537858 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:38.578204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:38.578149 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" podStartSLOduration=0.981914289 podStartE2EDuration="2.578134944s" totalImagesPullingTime="1.596220655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:38.577840456 +0000 UTC m=+294.444458623" watchObservedRunningTime="2026-10-30 13:08:38.578134944 +0000 UTC m=+294.444753111" Oct 30 13:08:44.548167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:44.548093 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:45.034252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.034218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.160459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.160426 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.491504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.491469 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-666b98648-qjmq6"] Oct 30 13:08:45.590848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.590812 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-666b98648-qjmq6"] Oct 30 13:08:45.591320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.590937 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.691490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.691458 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3921b04-5eab-4fef-ad91-326b111dd118-metrics-certs\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.691699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.691517 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3921b04-5eab-4fef-ad91-326b111dd118-maas-api-tls\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.691699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.691585 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3921b04-5eab-4fef-ad91-326b111dd118-openshift-service-ca\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.691787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.691760 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xc54\" (UniqueName: \"kubernetes.io/projected/c3921b04-5eab-4fef-ad91-326b111dd118-kube-api-access-4xc54\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.792715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.792630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xc54\" (UniqueName: \"kubernetes.io/projected/c3921b04-5eab-4fef-ad91-326b111dd118-kube-api-access-4xc54\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.792715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.792707 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3921b04-5eab-4fef-ad91-326b111dd118-metrics-certs\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.792945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.792739 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3921b04-5eab-4fef-ad91-326b111dd118-maas-api-tls\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.792945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.792755 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3921b04-5eab-4fef-ad91-326b111dd118-openshift-service-ca\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.793352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.793331 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3921b04-5eab-4fef-ad91-326b111dd118-openshift-service-ca\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.795113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.795092 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3921b04-5eab-4fef-ad91-326b111dd118-metrics-certs\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.795212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.795173 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3921b04-5eab-4fef-ad91-326b111dd118-maas-api-tls\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.811059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.811031 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xc54\" (UniqueName: \"kubernetes.io/projected/c3921b04-5eab-4fef-ad91-326b111dd118-kube-api-access-4xc54\") pod \"maas-api-666b98648-qjmq6\" (UID: \"c3921b04-5eab-4fef-ad91-326b111dd118\") " pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:45.938803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:45.938767 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:46.264611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:46.264559 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-666b98648-qjmq6"] Oct 30 13:08:46.268778 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:08:46.268747 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3921b04_5eab_4fef_ad91_326b111dd118.slice/crio-c645c82c3e41852a680481c178ea55f56248f056d4abed9c807b280c6c04be64 WatchSource:0}: Error finding container c645c82c3e41852a680481c178ea55f56248f056d4abed9c807b280c6c04be64: Status 404 returned error can't find the container with id c645c82c3e41852a680481c178ea55f56248f056d4abed9c807b280c6c04be64 Oct 30 13:08:46.574960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:46.574884 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" event={"ID":"c3921b04-5eab-4fef-ad91-326b111dd118","Type":"ContainerStarted","Data":"c645c82c3e41852a680481c178ea55f56248f056d4abed9c807b280c6c04be64"} Oct 30 13:08:47.580875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:47.580840 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" event={"ID":"c3921b04-5eab-4fef-ad91-326b111dd118","Type":"ContainerStarted","Data":"7d0f04619e56e692892dbbb6fe47cc26b266780640fe42208de23917fef2898a"} Oct 30 13:08:47.581283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:47.580962 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:47.620381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:47.620330 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" podStartSLOduration=2.248046611 podStartE2EDuration="2.620318806s" totalImagesPullingTime="372.272195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:47.620084947 +0000 UTC m=+303.486703121" watchObservedRunningTime="2026-10-30 13:08:47.620318806 +0000 UTC m=+303.486936978" Oct 30 13:08:51.334498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:51.334466 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:53.054211 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:08:53.054173 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:08:53.589983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:53.589953 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-666b98648-qjmq6" Oct 30 13:08:53.665003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:53.664971 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7666d54c64-x2std"] Oct 30 13:08:53.665215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:53.665189 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" podUID="ba780c5b-cbd2-48fd-b7d1-3679676a8e56" containerName="maas-api" containerID="cri-o://188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8" gracePeriod=30 Oct 30 13:08:53.905061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:53.905036 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:54.072911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.072871 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca\") pod \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " Oct 30 13:08:54.073380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.072919 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls\") pod \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " Oct 30 13:08:54.073380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.073017 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs\") pod \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " Oct 30 13:08:54.073380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.073042 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb\" (UniqueName: \"kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb\") pod \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\" (UID: \"ba780c5b-cbd2-48fd-b7d1-3679676a8e56\") " Oct 30 13:08:54.073380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.073305 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca" pod "ba780c5b-cbd2-48fd-b7d1-3679676a8e56" (UID: "ba780c5b-cbd2-48fd-b7d1-3679676a8e56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:54.075279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.075250 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls" pod "ba780c5b-cbd2-48fd-b7d1-3679676a8e56" (UID: "ba780c5b-cbd2-48fd-b7d1-3679676a8e56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:54.075412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.075270 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs" pod "ba780c5b-cbd2-48fd-b7d1-3679676a8e56" (UID: "ba780c5b-cbd2-48fd-b7d1-3679676a8e56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:54.075412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.075330 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb" pod "ba780c5b-cbd2-48fd-b7d1-3679676a8e56" (UID: "ba780c5b-cbd2-48fd-b7d1-3679676a8e56"). InnerVolumeSpecName "kube-api-access-hz5gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:54.173824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.173722 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.173824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.173761 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5gb\" (UniqueName: \"kubernetes.io/projected/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-kube-api-access-hz5gb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.173824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.173775 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.173824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.173789 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba780c5b-cbd2-48fd-b7d1-3679676a8e56-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.617317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.617275 5693 generic.go:356] "Generic (PLEG): container finished" podID="ba780c5b-cbd2-48fd-b7d1-3679676a8e56" containerID="188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8" exitCode=0 Oct 30 13:08:54.617317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.617321 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" event={"ID":"ba780c5b-cbd2-48fd-b7d1-3679676a8e56","Type":"ContainerDied","Data":"188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8"} Oct 30 13:08:54.617529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.617340 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" Oct 30 13:08:54.617529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.617387 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7666d54c64-x2std" event={"ID":"ba780c5b-cbd2-48fd-b7d1-3679676a8e56","Type":"ContainerDied","Data":"6b4047a3113ecf30991f838bead0eea58f51cb4ac1de8cd145a2e69a935e2bf2"} Oct 30 13:08:54.617529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.617409 5693 scope.go:122] "RemoveContainer" containerID="188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8" Oct 30 13:08:54.625837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.625815 5693 scope.go:122] "RemoveContainer" containerID="188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8" Oct 30 13:08:54.626105 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:08:54.626080 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8\": container with ID starting with 188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8 not found: ID does not exist" containerID="188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8" Oct 30 13:08:54.626167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.626114 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8"} err="failed to get container status \"188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8\": rpc error: code = NotFound desc = could not find container \"188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8\": container with ID starting with 188c8c9b41314e2fbb0788b078fbf513258436910d25edb5069ff61f00a71cd8 not found: ID does not exist" Oct 30 13:08:54.667243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.667218 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7666d54c64-x2std"] Oct 30 13:08:54.679235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:54.679209 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7666d54c64-x2std"] Oct 30 13:08:56.252457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:08:56.252424 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba780c5b-cbd2-48fd-b7d1-3679676a8e56" path="/var/lib/kubelet/pods/ba780c5b-cbd2-48fd-b7d1-3679676a8e56/volumes" Oct 30 13:09:23.052997 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:09:23.052960 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:09:51.433947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:09:51.433912 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:52.412176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:09:52.412142 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:52.544582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:09:52.544542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:53.054288 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:09:53.054243 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:10:02.749048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.748967 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4zc7z"] Oct 30 13:10:02.749426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.749377 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ba780c5b-cbd2-48fd-b7d1-3679676a8e56" containerName="maas-api" Oct 30 13:10:02.749426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.749395 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba780c5b-cbd2-48fd-b7d1-3679676a8e56" containerName="maas-api" Oct 30 13:10:02.749516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.749477 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba780c5b-cbd2-48fd-b7d1-3679676a8e56" containerName="maas-api" Oct 30 13:10:02.752584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.752569 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:02.778329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.778297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.785000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.784977 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rsbjl\"" Oct 30 13:10:02.785000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.784991 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 13:10:02.785149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.785001 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 13:10:02.786635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.786613 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4zc7z"] Oct 30 13:10:02.858653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.858617 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7xn\" (UniqueName: \"kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn\") pod \"ai-gateway-controller-7d85675f98-4zc7z\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:02.858833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.858685 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4zc7z\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:02.921329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.921293 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.959200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.959174 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4zc7z\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:02.959327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.959242 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7xn\" (UniqueName: \"kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn\") pod \"ai-gateway-controller-7d85675f98-4zc7z\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:02.961534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.961518 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4zc7z\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:02.979348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:02.979323 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7xn\" (UniqueName: \"kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn\") pod \"ai-gateway-controller-7d85675f98-4zc7z\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:03.065419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:03.065344 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:03.357058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:03.356959 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4zc7z"] Oct 30 13:10:03.361746 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:03.361719 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc20be9_a977_47b4_a381_2ce4176a9773.slice/crio-c3c54a1d51c4cddc0816f322c5857d1e38ed525bb3d654afb1801d425ca87286 WatchSource:0}: Error finding container c3c54a1d51c4cddc0816f322c5857d1e38ed525bb3d654afb1801d425ca87286: Status 404 returned error can't find the container with id c3c54a1d51c4cddc0816f322c5857d1e38ed525bb3d654afb1801d425ca87286 Oct 30 13:10:03.363025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:03.363005 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:10:03.909184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:03.909145 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" event={"ID":"3bc20be9-a977-47b4-a381-2ce4176a9773","Type":"ContainerStarted","Data":"c3c54a1d51c4cddc0816f322c5857d1e38ed525bb3d654afb1801d425ca87286"} Oct 30 13:10:05.919270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:05.919236 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" event={"ID":"3bc20be9-a977-47b4-a381-2ce4176a9773","Type":"ContainerStarted","Data":"692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94"} Oct 30 13:10:05.919672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:05.919375 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:05.957390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:05.957348 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" podStartSLOduration=2.420050077 podStartE2EDuration="3.957336476s" totalImagesPullingTime="1.537286399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:05.956204918 +0000 UTC m=+381.822823105" watchObservedRunningTime="2026-10-30 13:10:05.957336476 +0000 UTC m=+381.823954706" Oct 30 13:10:07.156911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.156880 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:07.224636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.224569 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-kg4wx"] Oct 30 13:10:07.228047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.228027 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.256198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.256174 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4h98p\"" Oct 30 13:10:07.256198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.256173 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 13:10:07.257568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.257544 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-kg4wx"] Oct 30 13:10:07.288899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.288867 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8trj2"] Oct 30 13:10:07.292464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.292443 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.298280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.298230 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:07.299097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.299074 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pscjk\" (UniqueName: \"kubernetes.io/projected/89c0aafb-9900-4eb3-90ec-d26f746bf51a-kube-api-access-pscjk\") pod \"payload-processing-5f78687d5b-kg4wx\" (UID: \"89c0aafb-9900-4eb3-90ec-d26f746bf51a\") " pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.299212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.299117 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c0aafb-9900-4eb3-90ec-d26f746bf51a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-kg4wx\" (UID: \"89c0aafb-9900-4eb3-90ec-d26f746bf51a\") " pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.309368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.309344 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8trj2"] Oct 30 13:10:07.399682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.399647 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79896ce2-39e6-406c-9079-e5077b542ad3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8trj2\" (UID: \"79896ce2-39e6-406c-9079-e5077b542ad3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.399853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.399775 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pscjk\" (UniqueName: \"kubernetes.io/projected/89c0aafb-9900-4eb3-90ec-d26f746bf51a-kube-api-access-pscjk\") pod \"payload-processing-5f78687d5b-kg4wx\" (UID: \"89c0aafb-9900-4eb3-90ec-d26f746bf51a\") " pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.399853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.399830 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c0aafb-9900-4eb3-90ec-d26f746bf51a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-kg4wx\" (UID: \"89c0aafb-9900-4eb3-90ec-d26f746bf51a\") " pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.399936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.399870 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5s7\" (UniqueName: \"kubernetes.io/projected/79896ce2-39e6-406c-9079-e5077b542ad3-kube-api-access-8k5s7\") pod \"payload-pre-processing-7b98bb678c-8trj2\" (UID: \"79896ce2-39e6-406c-9079-e5077b542ad3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.400496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.400479 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c0aafb-9900-4eb3-90ec-d26f746bf51a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-kg4wx\" (UID: \"89c0aafb-9900-4eb3-90ec-d26f746bf51a\") " pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.418541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.418475 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pscjk\" (UniqueName: \"kubernetes.io/projected/89c0aafb-9900-4eb3-90ec-d26f746bf51a-kube-api-access-pscjk\") pod \"payload-processing-5f78687d5b-kg4wx\" (UID: \"89c0aafb-9900-4eb3-90ec-d26f746bf51a\") " pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.501159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.501124 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5s7\" (UniqueName: \"kubernetes.io/projected/79896ce2-39e6-406c-9079-e5077b542ad3-kube-api-access-8k5s7\") pod \"payload-pre-processing-7b98bb678c-8trj2\" (UID: \"79896ce2-39e6-406c-9079-e5077b542ad3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.501159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.501162 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79896ce2-39e6-406c-9079-e5077b542ad3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8trj2\" (UID: \"79896ce2-39e6-406c-9079-e5077b542ad3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.501914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.501894 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79896ce2-39e6-406c-9079-e5077b542ad3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8trj2\" (UID: \"79896ce2-39e6-406c-9079-e5077b542ad3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.520304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.520276 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5s7\" (UniqueName: \"kubernetes.io/projected/79896ce2-39e6-406c-9079-e5077b542ad3-kube-api-access-8k5s7\") pod \"payload-pre-processing-7b98bb678c-8trj2\" (UID: \"79896ce2-39e6-406c-9079-e5077b542ad3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.541282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.541251 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:07.605078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.605045 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:07.835997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.835960 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-kg4wx"] Oct 30 13:10:07.839263 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:07.839233 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c0aafb_9900_4eb3_90ec_d26f746bf51a.slice/crio-52724b92b7fe171c3310ca75dd99797209b7d252ad410fe6b1adaa8a9a33b6f3 WatchSource:0}: Error finding container 52724b92b7fe171c3310ca75dd99797209b7d252ad410fe6b1adaa8a9a33b6f3: Status 404 returned error can't find the container with id 52724b92b7fe171c3310ca75dd99797209b7d252ad410fe6b1adaa8a9a33b6f3 Oct 30 13:10:07.902438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.901735 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8trj2"] Oct 30 13:10:07.908161 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:07.908137 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79896ce2_39e6_406c_9079_e5077b542ad3.slice/crio-cefc342f9aa8b5855445f2acb1dd24b133bb29d0e141d3d9ef7253927cad684b WatchSource:0}: Error finding container cefc342f9aa8b5855445f2acb1dd24b133bb29d0e141d3d9ef7253927cad684b: Status 404 returned error can't find the container with id cefc342f9aa8b5855445f2acb1dd24b133bb29d0e141d3d9ef7253927cad684b Oct 30 13:10:07.928592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.928561 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" event={"ID":"89c0aafb-9900-4eb3-90ec-d26f746bf51a","Type":"ContainerStarted","Data":"52724b92b7fe171c3310ca75dd99797209b7d252ad410fe6b1adaa8a9a33b6f3"} Oct 30 13:10:07.929642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:07.929620 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" event={"ID":"79896ce2-39e6-406c-9079-e5077b542ad3","Type":"ContainerStarted","Data":"cefc342f9aa8b5855445f2acb1dd24b133bb29d0e141d3d9ef7253927cad684b"} Oct 30 13:10:10.946871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:10.946827 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" event={"ID":"79896ce2-39e6-406c-9079-e5077b542ad3","Type":"ContainerStarted","Data":"2aca3e273f009cc0a3f3f12ebc5000c3adc8890158fa7ab3e8bca81494897d6c"} Oct 30 13:10:10.947328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:10.946934 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:10.948404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:10.948381 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" event={"ID":"89c0aafb-9900-4eb3-90ec-d26f746bf51a","Type":"ContainerStarted","Data":"2a0e4185ac5cc496719e0c7e22b391a53337e7c2699958da39f36cff840ed645"} Oct 30 13:10:10.948511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:10.948439 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:10.983927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:10.983883 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" podStartSLOduration=1.701515225 podStartE2EDuration="3.983871566s" totalImagesPullingTime="2.282356341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:10.983580673 +0000 UTC m=+386.850198859" watchObservedRunningTime="2026-10-30 13:10:10.983871566 +0000 UTC m=+386.850489717" Oct 30 13:10:11.021326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:11.021281 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" podStartSLOduration=1.703576926 podStartE2EDuration="4.021269959s" totalImagesPullingTime="2.317693033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:11.020958583 +0000 UTC m=+386.887576755" watchObservedRunningTime="2026-10-30 13:10:11.021269959 +0000 UTC m=+386.887888132" Oct 30 13:10:16.925495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:16.925461 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:17.262550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.262507 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-749c8d45cd-vlv69"] Oct 30 13:10:17.266366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.266344 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.285651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.285628 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-749c8d45cd-vlv69"] Oct 30 13:10:17.392352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.392316 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tm8w\" (UniqueName: \"kubernetes.io/projected/ab22081e-2b0e-419c-8f42-d9beaef9607e-kube-api-access-2tm8w\") pod \"ai-gateway-controller-749c8d45cd-vlv69\" (UID: \"ab22081e-2b0e-419c-8f42-d9beaef9607e\") " pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.392546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.392361 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab22081e-2b0e-419c-8f42-d9beaef9607e-metrics-certs\") pod \"ai-gateway-controller-749c8d45cd-vlv69\" (UID: \"ab22081e-2b0e-419c-8f42-d9beaef9607e\") " pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.493387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.493354 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tm8w\" (UniqueName: \"kubernetes.io/projected/ab22081e-2b0e-419c-8f42-d9beaef9607e-kube-api-access-2tm8w\") pod \"ai-gateway-controller-749c8d45cd-vlv69\" (UID: \"ab22081e-2b0e-419c-8f42-d9beaef9607e\") " pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.493569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.493398 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab22081e-2b0e-419c-8f42-d9beaef9607e-metrics-certs\") pod \"ai-gateway-controller-749c8d45cd-vlv69\" (UID: \"ab22081e-2b0e-419c-8f42-d9beaef9607e\") " pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.495874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.495853 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab22081e-2b0e-419c-8f42-d9beaef9607e-metrics-certs\") pod \"ai-gateway-controller-749c8d45cd-vlv69\" (UID: \"ab22081e-2b0e-419c-8f42-d9beaef9607e\") " pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.513031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.512975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tm8w\" (UniqueName: \"kubernetes.io/projected/ab22081e-2b0e-419c-8f42-d9beaef9607e-kube-api-access-2tm8w\") pod \"ai-gateway-controller-749c8d45cd-vlv69\" (UID: \"ab22081e-2b0e-419c-8f42-d9beaef9607e\") " pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.578966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.578936 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:17.875251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.875178 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-749c8d45cd-vlv69"] Oct 30 13:10:17.878615 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:17.878566 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab22081e_2b0e_419c_8f42_d9beaef9607e.slice/crio-034eb451890862ba10f93b82125200aff4ce0306a83b326bba9ef2e9a4f21d49 WatchSource:0}: Error finding container 034eb451890862ba10f93b82125200aff4ce0306a83b326bba9ef2e9a4f21d49: Status 404 returned error can't find the container with id 034eb451890862ba10f93b82125200aff4ce0306a83b326bba9ef2e9a4f21d49 Oct 30 13:10:17.979590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:17.979556 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" event={"ID":"ab22081e-2b0e-419c-8f42-d9beaef9607e","Type":"ContainerStarted","Data":"034eb451890862ba10f93b82125200aff4ce0306a83b326bba9ef2e9a4f21d49"} Oct 30 13:10:18.985430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:18.985386 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" event={"ID":"ab22081e-2b0e-419c-8f42-d9beaef9607e","Type":"ContainerStarted","Data":"4d0ef2585849a3c12b24c45bd622594a585e6fa47187945ae6c8f8700f3a5054"} Oct 30 13:10:18.985910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:18.985471 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:19.024344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:19.024297 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" podStartSLOduration=1.6107644589999999 podStartE2EDuration="2.02428282s" totalImagesPullingTime="413.518361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:19.023513742 +0000 UTC m=+394.890131928" watchObservedRunningTime="2026-10-30 13:10:19.02428282 +0000 UTC m=+394.890901040" Oct 30 13:10:21.955773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:21.955743 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8trj2" Oct 30 13:10:21.956170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:21.955811 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-kg4wx" Oct 30 13:10:23.054054 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:10:23.054015 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:10:29.992501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:29.992472 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-749c8d45cd-vlv69" Oct 30 13:10:30.063584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.063547 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4zc7z"] Oct 30 13:10:30.064074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.064034 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" podUID="3bc20be9-a977-47b4-a381-2ce4176a9773" containerName="manager" containerID="cri-o://692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94" gracePeriod=30 Oct 30 13:10:30.305710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.305670 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:30.413949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.413912 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn\" (UniqueName: \"kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn\") pod \"3bc20be9-a977-47b4-a381-2ce4176a9773\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " Oct 30 13:10:30.414112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.413976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs\") pod \"3bc20be9-a977-47b4-a381-2ce4176a9773\" (UID: \"3bc20be9-a977-47b4-a381-2ce4176a9773\") " Oct 30 13:10:30.416072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.416038 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn" pod "3bc20be9-a977-47b4-a381-2ce4176a9773" (UID: "3bc20be9-a977-47b4-a381-2ce4176a9773"). InnerVolumeSpecName "kube-api-access-sk7xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:30.416186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.416133 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs" pod "3bc20be9-a977-47b4-a381-2ce4176a9773" (UID: "3bc20be9-a977-47b4-a381-2ce4176a9773"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:30.515071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.515035 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7xn\" (UniqueName: \"kubernetes.io/projected/3bc20be9-a977-47b4-a381-2ce4176a9773-kube-api-access-sk7xn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:10:30.515071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:30.515066 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc20be9-a977-47b4-a381-2ce4176a9773-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:10:31.039803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.039763 5693 generic.go:356] "Generic (PLEG): container finished" podID="3bc20be9-a977-47b4-a381-2ce4176a9773" containerID="692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94" exitCode=0 Oct 30 13:10:31.040291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.039814 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" Oct 30 13:10:31.040291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.039849 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" event={"ID":"3bc20be9-a977-47b4-a381-2ce4176a9773","Type":"ContainerDied","Data":"692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94"} Oct 30 13:10:31.040291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.039906 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4zc7z" event={"ID":"3bc20be9-a977-47b4-a381-2ce4176a9773","Type":"ContainerDied","Data":"c3c54a1d51c4cddc0816f322c5857d1e38ed525bb3d654afb1801d425ca87286"} Oct 30 13:10:31.040291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.039932 5693 scope.go:122] "RemoveContainer" containerID="692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94" Oct 30 13:10:31.049560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.049540 5693 scope.go:122] "RemoveContainer" containerID="692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94" Oct 30 13:10:31.049832 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:10:31.049808 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94\": container with ID starting with 692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94 not found: ID does not exist" containerID="692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94" Oct 30 13:10:31.049919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.049848 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94"} err="failed to get container status \"692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94\": rpc error: code = NotFound desc = could not find container \"692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94\": container with ID starting with 692fe30c15c54df96705c0e0addcd8495f73d9b397d0963aa41bd890edc42e94 not found: ID does not exist" Oct 30 13:10:31.089945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.089911 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4zc7z"] Oct 30 13:10:31.101536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.101513 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4zc7z"] Oct 30 13:10:31.800538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:31.800501 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:32.252033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:32.251998 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc20be9-a977-47b4-a381-2ce4176a9773" path="/var/lib/kubelet/pods/3bc20be9-a977-47b4-a381-2ce4176a9773/volumes" Oct 30 13:10:32.303426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:32.303388 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:33.051213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:33.051182 5693 generic.go:356] "Generic (PLEG): container finished" podID="53c26842-64be-41e0-b367-57e23106974b" containerID="1fdc29a6f2448292944233825bf9293101ab743c2f7f7027274e3ed3d3df994d" exitCode=1 Oct 30 13:10:33.051394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:33.051255 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c7bfc749-57x8m" event={"ID":"53c26842-64be-41e0-b367-57e23106974b","Type":"ContainerDied","Data":"1fdc29a6f2448292944233825bf9293101ab743c2f7f7027274e3ed3d3df994d"} Oct 30 13:10:33.051663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:33.051643 5693 scope.go:122] "RemoveContainer" containerID="1fdc29a6f2448292944233825bf9293101ab743c2f7f7027274e3ed3d3df994d" Oct 30 13:10:33.578530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:33.578495 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:33.731120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:33.731083 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:34.058177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:34.058088 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c7bfc749-57x8m" event={"ID":"53c26842-64be-41e0-b367-57e23106974b","Type":"ContainerStarted","Data":"25665d3708595f4c98d8e6764833fcfb1e6999678b7f315c9652fd1fb5c9bc4f"} Oct 30 13:10:34.058345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:34.058314 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:10:45.065512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:45.065478 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78c7bfc749-57x8m" Oct 30 13:10:47.578005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:47.577970 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:47.726950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:47.726913 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:50.795177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.795138 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c"] Oct 30 13:10:50.795696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.795679 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3bc20be9-a977-47b4-a381-2ce4176a9773" containerName="manager" Oct 30 13:10:50.795760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.795697 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc20be9-a977-47b4-a381-2ce4176a9773" containerName="manager" Oct 30 13:10:50.795797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.795769 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc20be9-a977-47b4-a381-2ce4176a9773" containerName="manager" Oct 30 13:10:50.800212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.800185 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.828727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.828695 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 13:10:50.828879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.828829 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xkkzc\"" Oct 30 13:10:50.828879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.828848 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 13:10:50.829300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.829282 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 13:10:50.830178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.830156 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c"] Oct 30 13:10:50.891308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.891275 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.891453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.891310 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvb5r\" (UniqueName: \"kubernetes.io/projected/80b106a2-9b74-4de0-8ea8-9b11914f9f61-kube-api-access-tvb5r\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.891453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.891343 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b106a2-9b74-4de0-8ea8-9b11914f9f61-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.891527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.891470 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.891568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.891527 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.891645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.891618 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.920985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.920954 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:50.992617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.992573 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.992783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.992676 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.992783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.992702 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.992783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.992760 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.992963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.992789 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvb5r\" (UniqueName: \"kubernetes.io/projected/80b106a2-9b74-4de0-8ea8-9b11914f9f61-kube-api-access-tvb5r\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.992963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.992846 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b106a2-9b74-4de0-8ea8-9b11914f9f61-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.993068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.993014 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.993170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.993149 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.993170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.993158 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.995004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.994984 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b106a2-9b74-4de0-8ea8-9b11914f9f61-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:50.995187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:50.995166 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b106a2-9b74-4de0-8ea8-9b11914f9f61-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:51.011810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:51.011778 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvb5r\" (UniqueName: \"kubernetes.io/projected/80b106a2-9b74-4de0-8ea8-9b11914f9f61-kube-api-access-tvb5r\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c\" (UID: \"80b106a2-9b74-4de0-8ea8-9b11914f9f61\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:51.116794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:51.116715 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:51.416781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:51.416647 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c"] Oct 30 13:10:51.420791 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:51.420755 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b106a2_9b74_4de0_8ea8_9b11914f9f61.slice/crio-7f26c77a86db8e819f45e7ac4021ac8b4671f50d6ba78434f502aaafff58ccc8 WatchSource:0}: Error finding container 7f26c77a86db8e819f45e7ac4021ac8b4671f50d6ba78434f502aaafff58ccc8: Status 404 returned error can't find the container with id 7f26c77a86db8e819f45e7ac4021ac8b4671f50d6ba78434f502aaafff58ccc8 Oct 30 13:10:51.550004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:51.549968 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:52.142315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:52.142230 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" event={"ID":"80b106a2-9b74-4de0-8ea8-9b11914f9f61","Type":"ContainerStarted","Data":"7f26c77a86db8e819f45e7ac4021ac8b4671f50d6ba78434f502aaafff58ccc8"} Oct 30 13:10:53.055202 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:10:53.055162 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:10:53.148383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.148344 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" event={"ID":"80b106a2-9b74-4de0-8ea8-9b11914f9f61","Type":"ContainerStarted","Data":"be25afc59a363c304768a7cc0a0f46d119092ff2054159ce9c5c24eb92a55d2a"} Oct 30 13:10:53.190730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.190677 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" podStartSLOduration=2.007115837 podStartE2EDuration="3.190659484s" totalImagesPullingTime="1.183543647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:53.189865079 +0000 UTC m=+429.056483252" watchObservedRunningTime="2026-10-30 13:10:53.190659484 +0000 UTC m=+429.057277657" Oct 30 13:10:53.425937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.425861 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s"] Oct 30 13:10:53.430168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.430142 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.451113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.451083 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s"] Oct 30 13:10:53.460663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.460642 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 13:10:53.516087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.516053 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.516273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.516104 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.516273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.516194 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.516273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.516236 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd44a7d4-7135-435f-9041-fc4f6fde603c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.516273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.516261 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tlf\" (UniqueName: \"kubernetes.io/projected/dd44a7d4-7135-435f-9041-fc4f6fde603c-kube-api-access-q6tlf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.516478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.516404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.538521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.538497 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:53.616989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.616956 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617015 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617045 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617169 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd44a7d4-7135-435f-9041-fc4f6fde603c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617213 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tlf\" (UniqueName: \"kubernetes.io/projected/dd44a7d4-7135-435f-9041-fc4f6fde603c-kube-api-access-q6tlf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617331 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617426 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617268 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.617732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.617700 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.619652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.619632 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd44a7d4-7135-435f-9041-fc4f6fde603c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.620140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.620115 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd44a7d4-7135-435f-9041-fc4f6fde603c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.636814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.636786 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tlf\" (UniqueName: \"kubernetes.io/projected/dd44a7d4-7135-435f-9041-fc4f6fde603c-kube-api-access-q6tlf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rct2s\" (UID: \"dd44a7d4-7135-435f-9041-fc4f6fde603c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:53.743712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:53.743675 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:54.037216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:54.037132 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s"] Oct 30 13:10:54.040664 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:54.040632 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd44a7d4_7135_435f_9041_fc4f6fde603c.slice/crio-94ff03c4bb8310937bc81a99d61b234e34a8d79b2abe144af27fd59f9a27b0bc WatchSource:0}: Error finding container 94ff03c4bb8310937bc81a99d61b234e34a8d79b2abe144af27fd59f9a27b0bc: Status 404 returned error can't find the container with id 94ff03c4bb8310937bc81a99d61b234e34a8d79b2abe144af27fd59f9a27b0bc Oct 30 13:10:54.117705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:54.117674 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:10:54.154223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:54.154190 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" event={"ID":"dd44a7d4-7135-435f-9041-fc4f6fde603c","Type":"ContainerStarted","Data":"94ff03c4bb8310937bc81a99d61b234e34a8d79b2abe144af27fd59f9a27b0bc"} Oct 30 13:10:54.921514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:54.921480 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.159673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:55.159634 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" event={"ID":"dd44a7d4-7135-435f-9041-fc4f6fde603c","Type":"ContainerStarted","Data":"ba072cba66e0851c1d57720e1b7e4e83ea2e4dd5cd71696ab638bf312d8981f0"} Oct 30 13:10:55.198826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:55.198739 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" podStartSLOduration=1.997289169 podStartE2EDuration="2.198725911s" totalImagesPullingTime="201.436742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:55.197917117 +0000 UTC m=+431.064535290" watchObservedRunningTime="2026-10-30 13:10:55.198725911 +0000 UTC m=+431.065344081" Oct 30 13:10:55.744164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:55.744130 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:10:58.464403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.464364 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx"] Oct 30 13:10:58.468050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.468016 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.495178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.495154 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 13:10:58.496393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.496364 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx"] Oct 30 13:10:58.559936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.559899 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwz6\" (UniqueName: \"kubernetes.io/projected/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-kube-api-access-mlwz6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.560106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.559945 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.560156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.560115 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.560156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.560145 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.560234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.560205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.560282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.560268 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.580579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.580552 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.660722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.660683 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.660884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.660842 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.660958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.660886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwz6\" (UniqueName: \"kubernetes.io/projected/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-kube-api-access-mlwz6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.660958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.660916 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.661060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.661014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.661060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.661039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.661171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.661145 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.661234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.661182 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.661339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.661315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.663170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.663151 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.663677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.663656 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.680262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.680238 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwz6\" (UniqueName: \"kubernetes.io/projected/0cd02c9e-ae0b-4093-95a2-2ec4740b8055-kube-api-access-mlwz6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-h4jxx\" (UID: \"0cd02c9e-ae0b-4093-95a2-2ec4740b8055\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:58.785244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:58.785210 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:10:59.082560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:59.082464 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx"] Oct 30 13:10:59.086270 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:10:59.086240 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd02c9e_ae0b_4093_95a2_2ec4740b8055.slice/crio-1c63615bb37ce119f1ad9ce1f47af9c919325c2437c913cc06ff9e3a0080889a WatchSource:0}: Error finding container 1c63615bb37ce119f1ad9ce1f47af9c919325c2437c913cc06ff9e3a0080889a: Status 404 returned error can't find the container with id 1c63615bb37ce119f1ad9ce1f47af9c919325c2437c913cc06ff9e3a0080889a Oct 30 13:10:59.178866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:10:59.178832 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" event={"ID":"0cd02c9e-ae0b-4093-95a2-2ec4740b8055","Type":"ContainerStarted","Data":"1c63615bb37ce119f1ad9ce1f47af9c919325c2437c913cc06ff9e3a0080889a"} Oct 30 13:11:00.184420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:00.184382 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" event={"ID":"0cd02c9e-ae0b-4093-95a2-2ec4740b8055","Type":"ContainerStarted","Data":"cd3cb07d3ab8994d627b36a936992dc1d27b2a93649cf1d7e14e846119a66e57"} Oct 30 13:11:00.248030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:00.247995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:00.258411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:00.258367 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" podStartSLOduration=2.052426213 podStartE2EDuration="2.258352777s" totalImagesPullingTime="205.926564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:00.257001213 +0000 UTC m=+436.123619386" watchObservedRunningTime="2026-10-30 13:11:00.258352777 +0000 UTC m=+436.124970951" Oct 30 13:11:00.786300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:00.786265 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:11:01.117735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:01.117659 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:11:01.129961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:01.129935 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:11:01.199395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:01.199357 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c" Oct 30 13:11:02.974708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:02.974674 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bct5z"] Oct 30 13:11:02.978235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:02.978216 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:02.992801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:02.992779 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 13:11:03.006007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.005980 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bct5z"] Oct 30 13:11:03.097390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.097356 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.100509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.100484 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-home\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.100669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.100514 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.100669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.100540 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.100748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.100705 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.100785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.100756 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.100785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.100774 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwr6\" (UniqueName: \"kubernetes.io/projected/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-kube-api-access-qcwr6\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.201741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.201707 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.201925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.201752 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.201925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.201775 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwr6\" (UniqueName: \"kubernetes.io/projected/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-kube-api-access-qcwr6\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.201925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.201857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-home\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.201925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.201878 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.201925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.201911 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.202231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.202210 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-home\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.202298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.202275 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.202351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.202301 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.204221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.204197 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.204731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.204717 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.221457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.221432 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwr6\" (UniqueName: \"kubernetes.io/projected/0389eea9-e8d7-4d71-a2d2-378dc5ca820a-kube-api-access-qcwr6\") pod \"echo-simulator-kserve-64b6f8f96c-bct5z\" (UID: \"0389eea9-e8d7-4d71-a2d2-378dc5ca820a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.291888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.291808 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:03.591223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.591141 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bct5z"] Oct 30 13:11:03.594486 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:03.594460 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0389eea9_e8d7_4d71_a2d2_378dc5ca820a.slice/crio-a06e0dc42beb923a15c8b806acf631a203120bcca8ea3b38589c40be03856bd5 WatchSource:0}: Error finding container a06e0dc42beb923a15c8b806acf631a203120bcca8ea3b38589c40be03856bd5: Status 404 returned error can't find the container with id a06e0dc42beb923a15c8b806acf631a203120bcca8ea3b38589c40be03856bd5 Oct 30 13:11:03.744003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.743967 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:11:03.756112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:03.756084 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:11:04.205714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:04.205658 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" event={"ID":"0389eea9-e8d7-4d71-a2d2-378dc5ca820a","Type":"ContainerStarted","Data":"a06e0dc42beb923a15c8b806acf631a203120bcca8ea3b38589c40be03856bd5"} Oct 30 13:11:04.219104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:04.219077 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rct2s" Oct 30 13:11:04.650190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:04.647822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:06.216490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:06.216456 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" event={"ID":"0389eea9-e8d7-4d71-a2d2-378dc5ca820a","Type":"ContainerStarted","Data":"3be11b07629ad43c28c0dc64bae8180b6cc8a7e929654912037bbb774eafe80e"} Oct 30 13:11:06.257327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:06.257280 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" podStartSLOduration=2.603569915 podStartE2EDuration="4.257267849s" totalImagesPullingTime="1.653697934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:06.25673249 +0000 UTC m=+442.123350690" watchObservedRunningTime="2026-10-30 13:11:06.257267849 +0000 UTC m=+442.123886022" Oct 30 13:11:06.292696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:06.292666 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:06.464702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:06.464665 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:08.785834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:08.785788 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:11:08.798396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:08.798363 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:11:09.237480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.237450 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w"] Oct 30 13:11:09.241405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.241381 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.244221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.244198 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-h4jxx" Oct 30 13:11:09.264823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.264797 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cw6t5\"" Oct 30 13:11:09.287711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.287677 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w"] Oct 30 13:11:09.363126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363080 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363342 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363396 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363512 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363562 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh98z\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363655 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363763 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.363843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.363801 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464519 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464556 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464592 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464651 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464676 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464714 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464746 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.464799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464795 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh98z\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.465118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.464833 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.465118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.465006 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.465522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.465474 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.465522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.465513 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.465698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.465544 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.465698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.465513 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.467275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.467255 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.467357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.467302 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.484650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.484623 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.484769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.484650 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh98z\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z\") pod \"maas-default-gateway-openshift-default-67bf986f99-k7v2w\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.554537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.554462 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:09.876857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:09.876770 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w"] Oct 30 13:11:09.881882 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:09.881855 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e887d1_d90f_4a31_b99e_a72e3864802d.slice/crio-ff32b9edda46019030a1369a820aa19454c6e6475f69ff621fe21c9303065ded WatchSource:0}: Error finding container ff32b9edda46019030a1369a820aa19454c6e6475f69ff621fe21c9303065ded: Status 404 returned error can't find the container with id ff32b9edda46019030a1369a820aa19454c6e6475f69ff621fe21c9303065ded Oct 30 13:11:10.245755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:10.245715 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" event={"ID":"12e887d1-d90f-4a31-b99e-a72e3864802d","Type":"ContainerStarted","Data":"ff32b9edda46019030a1369a820aa19454c6e6475f69ff621fe21c9303065ded"} Oct 30 13:11:12.683479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:12.683444 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:12.683769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:12.683518 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:12.683769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:12.683549 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:13.264161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.264127 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" event={"ID":"12e887d1-d90f-4a31-b99e-a72e3864802d","Type":"ContainerStarted","Data":"9afe48ff829b9833582f429c6c5601a5094df481aa69ff516f581fd1c8f62859"} Oct 30 13:11:13.292703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.292673 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:13.305204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.305173 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:13.308986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.308941 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podStartSLOduration=1.510338197 podStartE2EDuration="4.308927807s" totalImagesPullingTime="2.79858961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:13.307024525 +0000 UTC m=+449.173642700" watchObservedRunningTime="2026-10-30 13:11:13.308927807 +0000 UTC m=+449.175545979" Oct 30 13:11:13.554732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.554635 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:13.556289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.556254 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:13.556421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:13.556315 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:14.279443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:14.279416 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bct5z" Oct 30 13:11:14.555525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:14.555430 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:14.555525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:14.555496 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:15.555165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:15.555127 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:15.555700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:15.555194 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:16.555784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:16.555745 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:16.556190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:16.555808 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:17.555081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:17.555040 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:17.555268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:17.555101 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:18.555319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:18.555275 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:18.555763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:18.555355 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:19.555293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:19.555256 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k7v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:11:19.555478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:19.555319 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:11:20.894499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:20.894465 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:20.895037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:20.895006 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:20.896100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:20.896079 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:11:23.022806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:23.022772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:23.054623 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:11:23.054569 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:11:23.192791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:23.192740 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:24.285788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.285753 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28"] Oct 30 13:11:24.292438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.292413 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.307380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.307347 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6"] Oct 30 13:11:24.311700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.311664 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28"] Oct 30 13:11:24.311874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.311825 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.335967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.335934 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6"] Oct 30 13:11:24.348592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348559 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcmq\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.348798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348640 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.348798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348676 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.348798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348721 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.348798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348761 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.349019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348799 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348824 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmt9\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348876 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.349019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348927 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.349019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.348956 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349002 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349040 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349135 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349170 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349209 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349247 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.349345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.349272 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450132 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450163 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450187 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450208 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450261 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450285 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcmq\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450312 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450340 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450378 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450418 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450448 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450471 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmt9\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450519 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450708 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450737 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.450950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.450740 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451024 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451043 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451067 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451264 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451274 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451355 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451431 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.451911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.451667 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.453312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.453290 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.453613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.453574 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.454140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.454121 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.454260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.454242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.469887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.469851 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcmq\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.474662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.474637 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k8pb6\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.474785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.474767 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.475065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.475042 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmt9\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9\") pod \"maas-default-gateway-openshift-default-67bf986f99-fkt28\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.606886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.606797 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:24.618789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.618759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:24.626664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.626642 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:24.877836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.877727 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:24.941879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.941725 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6"] Oct 30 13:11:24.944177 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:24.944148 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5f2cd5_06c6_4523_8cb7_3748c2fc0a06.slice/crio-9a8e68f3c1942f92107e9314a01632c6399a5b82bd53aa400c65f9113fd34d2a WatchSource:0}: Error finding container 9a8e68f3c1942f92107e9314a01632c6399a5b82bd53aa400c65f9113fd34d2a: Status 404 returned error can't find the container with id 9a8e68f3c1942f92107e9314a01632c6399a5b82bd53aa400c65f9113fd34d2a Oct 30 13:11:24.945062 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:24.945048 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eda9b4_c641_4abf_8db7_41086ce4b90f.slice/crio-89af882d93b5a1bfbe23b9242118ecc383c724080f8e739a29d414b17ff16383 WatchSource:0}: Error finding container 89af882d93b5a1bfbe23b9242118ecc383c724080f8e739a29d414b17ff16383: Status 404 returned error can't find the container with id 89af882d93b5a1bfbe23b9242118ecc383c724080f8e739a29d414b17ff16383 Oct 30 13:11:24.946825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.946794 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:24.946913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.946861 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:24.946913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.946894 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:24.947150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.947122 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:24.947239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.947194 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:24.947239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.947225 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:24.951636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:24.951615 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28"] Oct 30 13:11:25.322222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.322179 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" event={"ID":"f7eda9b4-c641-4abf-8db7-41086ce4b90f","Type":"ContainerStarted","Data":"7b43a6ee0702ba0afa8a310fe8188a29932686b0f5167bcfa00056dbb75a450b"} Oct 30 13:11:25.322842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.322241 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" event={"ID":"f7eda9b4-c641-4abf-8db7-41086ce4b90f","Type":"ContainerStarted","Data":"89af882d93b5a1bfbe23b9242118ecc383c724080f8e739a29d414b17ff16383"} Oct 30 13:11:25.323984 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.323961 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" event={"ID":"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06","Type":"ContainerStarted","Data":"89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee"} Oct 30 13:11:25.324099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.324001 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" event={"ID":"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06","Type":"ContainerStarted","Data":"9a8e68f3c1942f92107e9314a01632c6399a5b82bd53aa400c65f9113fd34d2a"} Oct 30 13:11:25.376581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.376455 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podStartSLOduration=1.376435822 podStartE2EDuration="1.376435822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:25.37420504 +0000 UTC m=+461.240823215" watchObservedRunningTime="2026-10-30 13:11:25.376435822 +0000 UTC m=+461.243053996" Oct 30 13:11:25.417641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.417558 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podStartSLOduration=1.417519355 podStartE2EDuration="1.417519355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:25.416296066 +0000 UTC m=+461.282914264" watchObservedRunningTime="2026-10-30 13:11:25.417519355 +0000 UTC m=+461.284137526" Oct 30 13:11:25.607171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.607091 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:25.630776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:25.630689 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:26.608783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:26.608748 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fkt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:11:26.609172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:26.608805 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:11:26.638623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:26.636125 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:11:26.638623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:26.636190 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:11:27.607676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:27.607639 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fkt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:11:27.607854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:27.607699 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:11:27.634623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:27.630904 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:11:27.634623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:27.630964 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:11:28.607498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:28.607468 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fkt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:11:28.607755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:28.607732 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:11:28.630919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:28.630890 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:11:28.631132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:28.631105 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:11:29.607298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:29.607259 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fkt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:11:29.607744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:29.607311 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:11:29.630997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:29.630960 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:11:29.631265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:29.631236 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:11:30.608988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:30.608950 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fkt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:11:30.609381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:30.609007 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:11:30.627531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:30.627498 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:11:30.627803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:30.627775 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:11:31.608054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:31.608012 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fkt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:11:31.608245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:31.608091 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:11:31.627217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:31.627178 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:11:31.627700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:31.627233 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:11:33.272720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:33.272684 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:33.273449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:33.273416 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:33.274226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:33.274203 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:33.274516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:33.274498 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:11:33.274616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:33.274552 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:33.275474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:33.275446 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:11:42.708398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:42.708357 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:42.855416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:42.855381 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:51.661255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:51.661221 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:53.054420 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:11:53.054383 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:11:53.573329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:53.573291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:53.778897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:53.778861 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.044778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.044746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.208002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.207967 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q"] Oct 30 13:11:54.218025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.217997 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.238473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.238445 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fc4197ef-openshift-default-dockercfg-8ppf7\"" Oct 30 13:11:54.246555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.246527 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.254932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.254905 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q"] Oct 30 13:11:54.278033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278000 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278057 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278180 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnv9\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278305 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278467 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278543 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.278677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.278661 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.370297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.370214 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z"] Oct 30 13:11:54.375048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.375023 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379132 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379169 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379230 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379272 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379334 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxl9\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379352 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379373 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379421 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379444 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnv9\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379462 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379506 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379523 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379573 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379589 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.379759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379617 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.379759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.380009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.379981 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.380096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.380079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.380325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.380303 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.380550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.380504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.380781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.380582 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.382318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.382293 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.383099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.383073 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.391902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.391877 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-662ed421-openshift-default-dockercfg-t2qxp\"" Oct 30 13:11:54.398925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.398899 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z"] Oct 30 13:11:54.409397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.409374 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.412095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.412071 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnv9\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9\") pod \"e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.414288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.414261 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.481050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481069 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481108 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481156 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481205 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxl9\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481232 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481331 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481370 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481620 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.481876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.481859 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.482126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.482109 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.484328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.484291 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.484741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.484709 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.486485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.486465 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.500685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.500652 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.500799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.500700 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxl9\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9\") pod \"e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.534584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.534557 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:54.573944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.573905 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.710651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.710593 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.711484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.711466 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:54.864795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.864763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.896370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.896207 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q"] Oct 30 13:11:54.899010 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:54.898982 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d865eba_0229_4aa7_aecf_8a251371ef74.slice/crio-3109b743d836552decffd70f5673e0554b29ed36024831ab8b23fee6a2a7fbec WatchSource:0}: Error finding container 3109b743d836552decffd70f5673e0554b29ed36024831ab8b23fee6a2a7fbec: Status 404 returned error can't find the container with id 3109b743d836552decffd70f5673e0554b29ed36024831ab8b23fee6a2a7fbec Oct 30 13:11:54.901258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.901228 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:54.901318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.901289 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:54.901350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:54.901317 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:55.019135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.019101 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z"] Oct 30 13:11:55.020747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.020715 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:55.023719 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:55.023697 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645c06d9_c5e7_41eb_ab62_88a69dd722d0.slice/crio-08d237e8a5cbbf7ddeffe1a97c9fbd714e81273d94c71cbf6d5a5954cf83a0ff WatchSource:0}: Error finding container 08d237e8a5cbbf7ddeffe1a97c9fbd714e81273d94c71cbf6d5a5954cf83a0ff: Status 404 returned error can't find the container with id 08d237e8a5cbbf7ddeffe1a97c9fbd714e81273d94c71cbf6d5a5954cf83a0ff Oct 30 13:11:55.026287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.026256 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:55.026374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.026321 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:55.026374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.026348 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:55.525505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.525462 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" event={"ID":"645c06d9-c5e7-41eb-ab62-88a69dd722d0","Type":"ContainerStarted","Data":"ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7"} Oct 30 13:11:55.525971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.525557 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" event={"ID":"645c06d9-c5e7-41eb-ab62-88a69dd722d0","Type":"ContainerStarted","Data":"08d237e8a5cbbf7ddeffe1a97c9fbd714e81273d94c71cbf6d5a5954cf83a0ff"} Oct 30 13:11:55.527384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.527353 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" event={"ID":"1d865eba-0229-4aa7-aecf-8a251371ef74","Type":"ContainerStarted","Data":"f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a"} Oct 30 13:11:55.527503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.527401 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" event={"ID":"1d865eba-0229-4aa7-aecf-8a251371ef74","Type":"ContainerStarted","Data":"3109b743d836552decffd70f5673e0554b29ed36024831ab8b23fee6a2a7fbec"} Oct 30 13:11:55.535715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.535698 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:55.540748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.540729 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:55.550772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.550740 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:55.567375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.567331 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podStartSLOduration=1.5673199169999998 podStartE2EDuration="1.567319917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:55.565960592 +0000 UTC m=+491.432578764" watchObservedRunningTime="2026-10-30 13:11:55.567319917 +0000 UTC m=+491.433938089" Oct 30 13:11:55.609120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.609073 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podStartSLOduration=1.609059535 podStartE2EDuration="1.609059535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:55.60853085 +0000 UTC m=+491.475149023" watchObservedRunningTime="2026-10-30 13:11:55.609059535 +0000 UTC m=+491.475677707" Oct 30 13:11:55.712465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.712411 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:55.938953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.938863 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4"] Oct 30 13:11:55.943545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.943519 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:55.962497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.962469 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-deb5d83f-openshift-default-dockercfg-fmb5z\"" Oct 30 13:11:55.965554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:55.965519 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4"] Oct 30 13:11:56.000715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.000676 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.000715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.000720 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.000947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.000793 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.000947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.000836 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.000947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.000870 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.001102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.001014 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzpsh\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.001137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.001110 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.001137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.001131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.001219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.001164 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.028679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.028645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.101699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101665 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzpsh\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.101885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101723 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.101885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101744 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.101885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101773 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.101885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101805 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.101885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.101885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101862 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101901 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.101939 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.102327 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.102415 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.102461 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.102624 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.102647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.102637 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.105023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.104994 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.105023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.104996 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.120729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.120704 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.120854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.120832 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzpsh\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh\") pod \"e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.167027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.166997 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.261724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.259060 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:56.533717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.533635 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:56.534639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.534568 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:11:56.591349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.589168 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4"] Oct 30 13:11:56.592065 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:56.591993 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32655306_fd82_4b28_aff7_4ce4a124e57e.slice/crio-056435dae9f55419571a4716b170cb9648e9728aecb157ab03e96d70a6056eae WatchSource:0}: Error finding container 056435dae9f55419571a4716b170cb9648e9728aecb157ab03e96d70a6056eae: Status 404 returned error can't find the container with id 056435dae9f55419571a4716b170cb9648e9728aecb157ab03e96d70a6056eae Oct 30 13:11:56.595046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.594997 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:56.595155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.595097 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:56.595155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.595144 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:56.718900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:56.718618 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:57.294225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.294192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.514393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.514362 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.540417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.540386 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" event={"ID":"32655306-fd82-4b28-aff7-4ce4a124e57e","Type":"ContainerStarted","Data":"0d507e440f21ba4ec4f916eff993aef966ff6285e1db721b093721fa7402e490"} Oct 30 13:11:57.540958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.540439 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" event={"ID":"32655306-fd82-4b28-aff7-4ce4a124e57e","Type":"ContainerStarted","Data":"056435dae9f55419571a4716b170cb9648e9728aecb157ab03e96d70a6056eae"} Oct 30 13:11:57.540958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.540842 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:57.541706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.541683 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:11:57.596672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.596564 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podStartSLOduration=2.596548848 podStartE2EDuration="2.596548848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:57.594585472 +0000 UTC m=+493.461203644" watchObservedRunningTime="2026-10-30 13:11:57.596548848 +0000 UTC m=+493.463167062" Oct 30 13:11:57.690975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.690943 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.977723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:57.977697 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:58.117577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.117543 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:58.259953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.259874 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:58.265223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.265198 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:58.546374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.546288 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:58.547006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.546986 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:11:58.684173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.684136 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8"] Oct 30 13:11:58.688403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.688381 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.709411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.709381 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d6a10701-serving-cert\"" Oct 30 13:11:58.712187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.712148 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8"] Oct 30 13:11:58.741756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.741703 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjhl\" (UniqueName: \"kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.741906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.741853 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.741906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.741897 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.742042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.742020 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.778148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.778116 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:58.843252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.843168 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjhl\" (UniqueName: \"kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.843252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.843235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.843491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.843271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.843491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.843340 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.844098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.844069 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.846234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.846209 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.846362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.846214 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.863808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.863778 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjhl\" (UniqueName: \"kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl\") pod \"maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:58.940824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:58.940791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:59.006407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.006334 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:11:59.017923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.017888 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc"] Oct 30 13:11:59.028123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.028069 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.040725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.040699 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc"] Oct 30 13:11:59.046640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.044003 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d6a10701\"" Oct 30 13:11:59.046640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.044063 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d6a10701-dockercfg-q5nxk\"" Oct 30 13:11:59.047271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.047245 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.047668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.047646 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9wp\" (UniqueName: \"kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp\") pod \"payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.094952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.094785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:59.149157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.149123 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.149312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.149199 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9wp\" (UniqueName: \"kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp\") pod \"payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.150077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.150051 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.168784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.168759 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9wp\" (UniqueName: \"kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp\") pod \"payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.351075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.350992 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8"] Oct 30 13:11:59.355525 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:59.355498 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72af4e9_33c9_4d64_ac6a_821be772a350.slice/crio-c8fd57df97d8bbd03f37395e42c99e26d2dfb2c2f3e939b142e2cc1620d839e4 WatchSource:0}: Error finding container c8fd57df97d8bbd03f37395e42c99e26d2dfb2c2f3e939b142e2cc1620d839e4: Status 404 returned error can't find the container with id c8fd57df97d8bbd03f37395e42c99e26d2dfb2c2f3e939b142e2cc1620d839e4 Oct 30 13:11:59.380155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.380132 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:11:59.552418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.552383 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" event={"ID":"a72af4e9-33c9-4d64-ac6a-821be772a350","Type":"ContainerStarted","Data":"c8fd57df97d8bbd03f37395e42c99e26d2dfb2c2f3e939b142e2cc1620d839e4"} Oct 30 13:11:59.675849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:11:59.675764 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc"] Oct 30 13:11:59.679877 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:11:59.679851 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaaf2c9_2166_428a_8607_3b5caa50f76e.slice/crio-31d3e1c4bd10c54c53f0794bbaa888b521055c1d7fef9720535c31ab3f0b78f6 WatchSource:0}: Error finding container 31d3e1c4bd10c54c53f0794bbaa888b521055c1d7fef9720535c31ab3f0b78f6: Status 404 returned error can't find the container with id 31d3e1c4bd10c54c53f0794bbaa888b521055c1d7fef9720535c31ab3f0b78f6 Oct 30 13:12:00.262735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.262705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:00.426292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.426254 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:00.559215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.559125 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" event={"ID":"a72af4e9-33c9-4d64-ac6a-821be772a350","Type":"ContainerStarted","Data":"19e174aea08d52139df15698c9e17c3623ae30921ecede19d7706479e482b079"} Oct 30 13:12:00.559767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.559338 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:12:00.561496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.561465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" event={"ID":"1eaaf2c9-2166-428a-8607-3b5caa50f76e","Type":"ContainerStarted","Data":"08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b"} Oct 30 13:12:00.561642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.561527 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" event={"ID":"1eaaf2c9-2166-428a-8607-3b5caa50f76e","Type":"ContainerStarted","Data":"31d3e1c4bd10c54c53f0794bbaa888b521055c1d7fef9720535c31ab3f0b78f6"} Oct 30 13:12:00.561766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.561749 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:12:00.625350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.625280 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" podStartSLOduration=2.151851609 podStartE2EDuration="2.625261472s" totalImagesPullingTime="473.409863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:00.622940327 +0000 UTC m=+496.489558511" watchObservedRunningTime="2026-10-30 13:12:00.625261472 +0000 UTC m=+496.491879649" Oct 30 13:12:00.645754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.645724 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:00.686240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.686187 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" podStartSLOduration=2.686173472 podStartE2EDuration="2.686173472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:00.684756365 +0000 UTC m=+496.551374538" watchObservedRunningTime="2026-10-30 13:12:00.686173472 +0000 UTC m=+496.552791645" Oct 30 13:12:00.899524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:00.899443 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.069918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.069881 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.250334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.250289 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.289927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.289898 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b"] Oct 30 13:12:01.293934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.293911 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.314530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.314504 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b"] Oct 30 13:12:01.315880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.315854 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc4197ef\"" Oct 30 13:12:01.382192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.382158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wt6\" (UniqueName: \"kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6\") pod \"payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.382350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.382299 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.408181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.408144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.484247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.484202 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wt6\" (UniqueName: \"kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6\") pod \"payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.484481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.484317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.485445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.485417 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.503130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.503065 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wt6\" (UniqueName: \"kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6\") pod \"payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.606832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.606796 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:01.924657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:01.924565 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b"] Oct 30 13:12:01.928482 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:01.928453 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8149c5e4_2210_436c_be84_b941291a268f.slice/crio-a3d69ee5a0bddf840a96b03338f5aa2c585f2f97b8d4997234ac8ab0dc7c9320 WatchSource:0}: Error finding container a3d69ee5a0bddf840a96b03338f5aa2c585f2f97b8d4997234ac8ab0dc7c9320: Status 404 returned error can't find the container with id a3d69ee5a0bddf840a96b03338f5aa2c585f2f97b8d4997234ac8ab0dc7c9320 Oct 30 13:12:02.573676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.573635 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" event={"ID":"8149c5e4-2210-436c-be84-b941291a268f","Type":"ContainerStarted","Data":"25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7"} Oct 30 13:12:02.573972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.573719 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" event={"ID":"8149c5e4-2210-436c-be84-b941291a268f","Type":"ContainerStarted","Data":"a3d69ee5a0bddf840a96b03338f5aa2c585f2f97b8d4997234ac8ab0dc7c9320"} Oct 30 13:12:02.573972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.573745 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:02.614993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.614934 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" podStartSLOduration=1.6149204419999998 podStartE2EDuration="1.614920442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:02.613325022 +0000 UTC m=+498.479943196" watchObservedRunningTime="2026-10-30 13:12:02.614920442 +0000 UTC m=+498.481538614" Oct 30 13:12:02.755025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.754984 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f"] Oct 30 13:12:02.758570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.758547 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.776997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.776969 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f"] Oct 30 13:12:02.792044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.792017 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fc4197ef-serving-cert\"" Oct 30 13:12:02.814363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.814336 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.814519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.814369 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.814649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.814622 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.814734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.814716 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7pr\" (UniqueName: \"kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.819738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.819707 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:02.915717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.915628 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.915717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.915667 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.915947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.915764 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.915947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.915797 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7pr\" (UniqueName: \"kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.915947 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:02.915804 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-serving-cert: secret "maas-api-e2e-ait-fc4197ef-serving-cert" not found Oct 30 13:12:02.915947 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:02.915886 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls podName:68896e1d-68b8-42be-b3c3-19b4c9ae166e nodeName:}" failed. No retries permitted until 2026-10-30 13:12:03.41586409 +0000 UTC m=+499.282482260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls") pod "maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" (UID: "68896e1d-68b8-42be-b3c3-19b4c9ae166e") : secret "maas-api-e2e-ait-fc4197ef-serving-cert" not found Oct 30 13:12:02.916445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.916418 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.918246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.918227 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.935578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.935544 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7pr\" (UniqueName: \"kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:02.984310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:02.984260 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:03.141796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.141759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:03.330458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.330428 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:03.421130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.421093 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:03.423861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.423818 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\") pod \"maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:03.489410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.489367 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:03.558613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.557923 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh"] Oct 30 13:12:03.570929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.570015 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.574451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.573251 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh"] Oct 30 13:12:03.589764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.589115 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-deb5d83f\"" Oct 30 13:12:03.589764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.589184 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-deb5d83f-dockercfg-r7zw6\"" Oct 30 13:12:03.629841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.629808 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume\") pod \"payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.630279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.629907 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwhf\" (UniqueName: \"kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf\") pod \"payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.672433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.672382 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:03.731303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.731273 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwhf\" (UniqueName: \"kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf\") pod \"payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.731425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.731403 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume\") pod \"payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.732254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.732231 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume\") pod \"payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.751819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.751793 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwhf\" (UniqueName: \"kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf\") pod \"payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.809797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.809765 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:03.888686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.887123 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:03.994428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:03.994274 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f"] Oct 30 13:12:04.009794 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:04.009754 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68896e1d_68b8_42be_b3c3_19b4c9ae166e.slice/crio-001a2f7a3edd604b8bd89d3b1c5e5e81a9a71fc1fa74be4b35d79bb8cfd3a073 WatchSource:0}: Error finding container 001a2f7a3edd604b8bd89d3b1c5e5e81a9a71fc1fa74be4b35d79bb8cfd3a073: Status 404 returned error can't find the container with id 001a2f7a3edd604b8bd89d3b1c5e5e81a9a71fc1fa74be4b35d79bb8cfd3a073 Oct 30 13:12:04.026151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.026077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:04.183955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.183875 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh"] Oct 30 13:12:04.187567 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:04.187534 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327aa54a_867d_4c9e_ab72_315010ccaa52.slice/crio-495e1bbf7971d785f08452dfcc53852484e92cdfa84e2b9b8d363a98bb358ab9 WatchSource:0}: Error finding container 495e1bbf7971d785f08452dfcc53852484e92cdfa84e2b9b8d363a98bb358ab9: Status 404 returned error can't find the container with id 495e1bbf7971d785f08452dfcc53852484e92cdfa84e2b9b8d363a98bb358ab9 Oct 30 13:12:04.587955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.587919 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" event={"ID":"327aa54a-867d-4c9e-ab72-315010ccaa52","Type":"ContainerStarted","Data":"66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5"} Oct 30 13:12:04.588155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.587978 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" event={"ID":"327aa54a-867d-4c9e-ab72-315010ccaa52","Type":"ContainerStarted","Data":"495e1bbf7971d785f08452dfcc53852484e92cdfa84e2b9b8d363a98bb358ab9"} Oct 30 13:12:04.588155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.588129 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:04.589655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.589628 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" event={"ID":"68896e1d-68b8-42be-b3c3-19b4c9ae166e","Type":"ContainerStarted","Data":"55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa"} Oct 30 13:12:04.589774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.589677 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" event={"ID":"68896e1d-68b8-42be-b3c3-19b4c9ae166e","Type":"ContainerStarted","Data":"001a2f7a3edd604b8bd89d3b1c5e5e81a9a71fc1fa74be4b35d79bb8cfd3a073"} Oct 30 13:12:04.589774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.589707 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:04.631036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.628275 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" podStartSLOduration=1.6282574589999999 podStartE2EDuration="1.628257459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:04.624775323 +0000 UTC m=+500.491393497" watchObservedRunningTime="2026-10-30 13:12:04.628257459 +0000 UTC m=+500.494875633" Oct 30 13:12:04.664482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.664427 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" podStartSLOduration=2.256720079 podStartE2EDuration="2.664412509s" totalImagesPullingTime="407.69243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:04.662448472 +0000 UTC m=+500.529066645" watchObservedRunningTime="2026-10-30 13:12:04.664412509 +0000 UTC m=+500.531030706" Oct 30 13:12:04.821630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:04.815094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.242583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.242548 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7"] Oct 30 13:12:06.247019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.246995 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.261625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.261577 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-deb5d83f-serving-cert\"" Oct 30 13:12:06.276743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.276710 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7"] Oct 30 13:12:06.364096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.364059 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.364267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.364169 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6kh\" (UniqueName: \"kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.364267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.364226 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.364348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.364294 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.373274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.373243 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.465172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.465144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.465362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.465237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6kh\" (UniqueName: \"kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.465362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.465276 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.465362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.465357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.466077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.466051 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.467817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.467797 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.467817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.467807 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.483836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.483815 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6kh\" (UniqueName: \"kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh\") pod \"maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.528690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.510019 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.562851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.562815 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:06.573259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.573231 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:12:06.653525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.653498 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.865858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:06.865774 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7"] Oct 30 13:12:06.869332 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:06.869294 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1210f0cc_9e7d_4eda_b076_cd73b0470d4b.slice/crio-0fb0adcdb193fe16222675710e648f568d2738d9a273f698ab0e2623486b3969 WatchSource:0}: Error finding container 0fb0adcdb193fe16222675710e648f568d2738d9a273f698ab0e2623486b3969: Status 404 returned error can't find the container with id 0fb0adcdb193fe16222675710e648f568d2738d9a273f698ab0e2623486b3969 Oct 30 13:12:07.006210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:07.006170 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:07.166804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:07.166710 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:07.608030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:07.607993 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" event={"ID":"1210f0cc-9e7d-4eda-b076-cd73b0470d4b","Type":"ContainerStarted","Data":"62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878"} Oct 30 13:12:07.608410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:07.608049 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" event={"ID":"1210f0cc-9e7d-4eda-b076-cd73b0470d4b","Type":"ContainerStarted","Data":"0fb0adcdb193fe16222675710e648f568d2738d9a273f698ab0e2623486b3969"} Oct 30 13:12:07.608410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:07.608108 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:07.651542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:07.651487 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" podStartSLOduration=1.239881681 podStartE2EDuration="1.651472353s" totalImagesPullingTime="411.590672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:07.650108785 +0000 UTC m=+503.516726977" watchObservedRunningTime="2026-10-30 13:12:07.651472353 +0000 UTC m=+503.518090549" Oct 30 13:12:08.432909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:08.420193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:09.149856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:09.149704 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:09.312220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:09.312177 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:10.524337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:10.524304 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:10.604395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:10.604364 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:11.578060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:11.578029 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:12:12.703441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:12.703408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.602812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:13.602772 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:13.619345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:13.619316 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:14.860844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:14.860807 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:15.599803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:15.599771 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:16.669797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.669750 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4"] Oct 30 13:12:16.675416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.675392 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.680496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.680458 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544rs\" (UniqueName: \"kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.680672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.680561 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.680672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.680629 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.680672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.680656 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.692781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.692756 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fb3b54cb-serving-cert\"" Oct 30 13:12:16.711354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.711317 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4"] Oct 30 13:12:16.781347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.781305 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.781551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.781361 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.781551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.781383 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.781551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.781454 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544rs\" (UniqueName: \"kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.781912 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:16.781884 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-serving-cert: secret "maas-api-e2e-derive-fb3b54cb-serving-cert" not found Oct 30 13:12:16.782035 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:16.781971 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls podName:6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0 nodeName:}" failed. No retries permitted until 2026-10-30 13:12:17.281948472 +0000 UTC m=+513.148566647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls") pod "maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" (UID: "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0") : secret "maas-api-e2e-derive-fb3b54cb-serving-cert" not found Oct 30 13:12:16.782410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.782388 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.784186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.784163 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.803423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.803392 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544rs\" (UniqueName: \"kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:16.804392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.804367 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.956373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:16.956285 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.096964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:17.096927 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.287746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:17.287711 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:17.290329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:17.290296 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\") pod \"maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:17.589734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:17.589649 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:17.924041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:17.923948 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4"] Oct 30 13:12:17.956208 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:17.956165 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3e55b3_8cae_4d2e_a3b9_be9c9a5882d0.slice/crio-cac0a14abdf9bbc61db90a060fb171248532378c9c05f794fceb32e789338b72 WatchSource:0}: Error finding container cac0a14abdf9bbc61db90a060fb171248532378c9c05f794fceb32e789338b72: Status 404 returned error can't find the container with id cac0a14abdf9bbc61db90a060fb171248532378c9c05f794fceb32e789338b72 Oct 30 13:12:18.781623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:18.780808 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" event={"ID":"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0","Type":"ContainerStarted","Data":"9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca"} Oct 30 13:12:18.781623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:18.780951 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" event={"ID":"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0","Type":"ContainerStarted","Data":"cac0a14abdf9bbc61db90a060fb171248532378c9c05f794fceb32e789338b72"} Oct 30 13:12:18.781623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:18.780997 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:18.822316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:18.822222 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" podStartSLOduration=2.492144224 podStartE2EDuration="2.822170744s" totalImagesPullingTime="330.02652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:18.820373092 +0000 UTC m=+514.686991284" watchObservedRunningTime="2026-10-30 13:12:18.822170744 +0000 UTC m=+514.688788919" Oct 30 13:12:18.858025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:18.857990 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.789172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.789132 5693 generic.go:356] "Generic (PLEG): container finished" podID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerID="ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7" exitCode=0 Oct 30 13:12:19.789708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.789217 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" event={"ID":"645c06d9-c5e7-41eb-ab62-88a69dd722d0","Type":"ContainerDied","Data":"ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7"} Oct 30 13:12:19.789827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.789800 5693 scope.go:122] "RemoveContainer" containerID="ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7" Oct 30 13:12:19.791195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.791173 5693 generic.go:356] "Generic (PLEG): container finished" podID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerID="f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a" exitCode=0 Oct 30 13:12:19.791270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.791246 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" event={"ID":"1d865eba-0229-4aa7-aecf-8a251371ef74","Type":"ContainerDied","Data":"f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a"} Oct 30 13:12:19.791525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.791492 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:19.791623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.791591 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:19.791680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.791663 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:19.791715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.791698 5693 scope.go:122] "RemoveContainer" containerID="f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a" Oct 30 13:12:19.793006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.792974 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:19.793088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.793054 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:19.793127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:19.793103 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:20.798629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:20.798447 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" event={"ID":"645c06d9-c5e7-41eb-ab62-88a69dd722d0","Type":"ContainerStarted","Data":"4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc"} Oct 30 13:12:20.801523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:20.801494 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" event={"ID":"1d865eba-0229-4aa7-aecf-8a251371ef74","Type":"ContainerStarted","Data":"a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6"} Oct 30 13:12:21.060767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.056166 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.541930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.536006 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:12:21.541930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.536147 5693 patch_prober.go:29] interesting pod/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:12:21.541930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.536190 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:12:21.712008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.711969 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:12:21.712550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.712525 5693 patch_prober.go:29] interesting pod/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:12:21.712666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:21.712584 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:12:22.542624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:22.537026 5693 patch_prober.go:29] interesting pod/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:12:22.542624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:22.537095 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:12:22.711899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:22.711861 5693 patch_prober.go:29] interesting pod/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:12:22.712075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:22.711919 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:12:23.021668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.021632 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.123859 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:23.123823 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:12:23.417824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.417746 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc"] Oct 30 13:12:23.552109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.551266 5693 patch_prober.go:29] interesting pod/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:12:23.552109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.551337 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:12:23.601615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.601566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.712617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.712515 5693 patch_prober.go:29] interesting pod/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:12:23.712807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.712647 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:12:23.757268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:23.757232 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:24.538623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.535003 5693 patch_prober.go:29] interesting pod/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:12:24.538623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.535071 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:12:24.712874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.712791 5693 patch_prober.go:29] interesting pod/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:12:24.713330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.713299 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:12:24.796433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.796402 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:24.876581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.876543 5693 generic.go:356] "Generic (PLEG): container finished" podID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerID="0d507e440f21ba4ec4f916eff993aef966ff6285e1db721b093721fa7402e490" exitCode=0 Oct 30 13:12:24.877003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.876977 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" event={"ID":"32655306-fd82-4b28-aff7-4ce4a124e57e","Type":"ContainerDied","Data":"0d507e440f21ba4ec4f916eff993aef966ff6285e1db721b093721fa7402e490"} Oct 30 13:12:24.877632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.877591 5693 scope.go:122] "RemoveContainer" containerID="0d507e440f21ba4ec4f916eff993aef966ff6285e1db721b093721fa7402e490" Oct 30 13:12:24.888757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.887867 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:24.889527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.889501 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:24.889760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:24.889735 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:25.449468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:25.449431 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.535141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:25.535105 5693 patch_prober.go:29] interesting pod/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:12:25.535454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:25.535407 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:12:25.724028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:25.723989 5693 patch_prober.go:29] interesting pod/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:12:25.724517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:25.724051 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:12:25.905624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:25.905138 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" event={"ID":"32655306-fd82-4b28-aff7-4ce4a124e57e","Type":"ContainerStarted","Data":"a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f"} Oct 30 13:12:26.260838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.260800 5693 patch_prober.go:29] interesting pod/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:12:26.274710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.274667 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:12:26.274944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.271143 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:12:26.546682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.535824 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:12:26.552620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.548893 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:12:26.716556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.716531 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:12:26.716996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.716978 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:12:26.717732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.717712 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:12:26.911826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:26.911751 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:12:27.259698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:27.259660 5693 patch_prober.go:29] interesting pod/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:12:27.259872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:27.259730 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:12:27.822893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:27.822861 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:28.259909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.259868 5693 patch_prober.go:29] interesting pod/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:12:28.260311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.259931 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:12:28.418127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.418074 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" podUID="1eaaf2c9-2166-428a-8607-3b5caa50f76e" containerName="payload-processing" containerID="cri-o://08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b" gracePeriod=25 Oct 30 13:12:28.547729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.546724 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:12:28.754972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.754939 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:12:28.862508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.862427 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume\") pod \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " Oct 30 13:12:28.862928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.862904 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp\" (UniqueName: \"kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp\") pod \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\" (UID: \"1eaaf2c9-2166-428a-8607-3b5caa50f76e\") " Oct 30 13:12:28.863340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.863311 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume" pod "1eaaf2c9-2166-428a-8607-3b5caa50f76e" (UID: "1eaaf2c9-2166-428a-8607-3b5caa50f76e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:28.863553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.863537 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaaf2c9-2166-428a-8607-3b5caa50f76e-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:28.865960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.865938 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp" pod "1eaaf2c9-2166-428a-8607-3b5caa50f76e" (UID: "1eaaf2c9-2166-428a-8607-3b5caa50f76e"). InnerVolumeSpecName "kube-api-access-hn9wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:28.924281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.924243 5693 generic.go:356] "Generic (PLEG): container finished" podID="1eaaf2c9-2166-428a-8607-3b5caa50f76e" containerID="08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b" exitCode=0 Oct 30 13:12:28.924486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.924294 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" Oct 30 13:12:28.924486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.924346 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" event={"ID":"1eaaf2c9-2166-428a-8607-3b5caa50f76e","Type":"ContainerDied","Data":"08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b"} Oct 30 13:12:28.924585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.924503 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc" event={"ID":"1eaaf2c9-2166-428a-8607-3b5caa50f76e","Type":"ContainerDied","Data":"31d3e1c4bd10c54c53f0794bbaa888b521055c1d7fef9720535c31ab3f0b78f6"} Oct 30 13:12:28.924585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.924554 5693 scope.go:122] "RemoveContainer" containerID="08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b" Oct 30 13:12:28.956318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.956279 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b"] Oct 30 13:12:28.964950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.964925 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9wp\" (UniqueName: \"kubernetes.io/projected/1eaaf2c9-2166-428a-8607-3b5caa50f76e-kube-api-access-hn9wp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:28.968620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.968570 5693 scope.go:122] "RemoveContainer" containerID="08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b" Oct 30 13:12:28.968961 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:28.968942 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b\": container with ID starting with 08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b not found: ID does not exist" containerID="08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b" Oct 30 13:12:28.969031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:28.968970 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b"} err="failed to get container status \"08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b\": rpc error: code = NotFound desc = could not find container \"08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b\": container with ID starting with 08afe670db610dc02576119c52be102c8968a2a786a210cdf15ef1eca16c321b not found: ID does not exist" Oct 30 13:12:29.063394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:29.061488 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc"] Oct 30 13:12:29.063394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:29.063317 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d6a10701-7d7547748d-5rscc"] Oct 30 13:12:29.260294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:29.260256 5693 patch_prober.go:29] interesting pod/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:12:29.260809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:29.260307 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:12:29.421340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:29.421236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.556526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:29.556104 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:30.257964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:30.257932 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eaaf2c9-2166-428a-8607-3b5caa50f76e" path="/var/lib/kubelet/pods/1eaaf2c9-2166-428a-8607-3b5caa50f76e/volumes" Oct 30 13:12:30.259639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:30.259615 5693 patch_prober.go:29] interesting pod/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:12:30.259784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:30.259666 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:12:31.260359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:31.260323 5693 patch_prober.go:29] interesting pod/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:12:31.260797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:31.260386 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:12:31.415961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:31.414932 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:32.264757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.264730 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:12:32.265699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.265680 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:12:32.839307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.839264 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8"] Oct 30 13:12:32.839879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.839838 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" podUID="a72af4e9-33c9-4d64-ac6a-821be772a350" containerName="maas-api" containerID="cri-o://19e174aea08d52139df15698c9e17c3623ae30921ecede19d7706479e482b079" gracePeriod=30 Oct 30 13:12:32.942619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.942570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:32.979461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.979427 5693 generic.go:356] "Generic (PLEG): container finished" podID="a72af4e9-33c9-4d64-ac6a-821be772a350" containerID="19e174aea08d52139df15698c9e17c3623ae30921ecede19d7706479e482b079" exitCode=0 Oct 30 13:12:32.979656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:32.979496 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" event={"ID":"a72af4e9-33c9-4d64-ac6a-821be772a350","Type":"ContainerDied","Data":"19e174aea08d52139df15698c9e17c3623ae30921ecede19d7706479e482b079"} Oct 30 13:12:33.136765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.136737 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:12:33.218895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.218858 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca\") pod \"a72af4e9-33c9-4d64-ac6a-821be772a350\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " Oct 30 13:12:33.219056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.218902 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl\" (UniqueName: \"kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl\") pod \"a72af4e9-33c9-4d64-ac6a-821be772a350\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " Oct 30 13:12:33.219113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.219080 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs\") pod \"a72af4e9-33c9-4d64-ac6a-821be772a350\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " Oct 30 13:12:33.219183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.219143 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls\") pod \"a72af4e9-33c9-4d64-ac6a-821be772a350\" (UID: \"a72af4e9-33c9-4d64-ac6a-821be772a350\") " Oct 30 13:12:33.219290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.219259 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca" pod "a72af4e9-33c9-4d64-ac6a-821be772a350" (UID: "a72af4e9-33c9-4d64-ac6a-821be772a350"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:33.219658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.219633 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72af4e9-33c9-4d64-ac6a-821be772a350-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:33.221221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.221194 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs" pod "a72af4e9-33c9-4d64-ac6a-821be772a350" (UID: "a72af4e9-33c9-4d64-ac6a-821be772a350"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:33.221221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.221198 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl" pod "a72af4e9-33c9-4d64-ac6a-821be772a350" (UID: "a72af4e9-33c9-4d64-ac6a-821be772a350"). InnerVolumeSpecName "kube-api-access-9fjhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:33.221401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.221323 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls" pod "a72af4e9-33c9-4d64-ac6a-821be772a350" (UID: "a72af4e9-33c9-4d64-ac6a-821be772a350"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:33.320943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.320913 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:33.320943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.320939 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72af4e9-33c9-4d64-ac6a-821be772a350-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:33.321327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.320950 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjhl\" (UniqueName: \"kubernetes.io/projected/a72af4e9-33c9-4d64-ac6a-821be772a350-kube-api-access-9fjhl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:33.496280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.496235 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.646008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.645971 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.793735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.793659 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.931766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.931730 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.957373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.957319 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" podUID="8149c5e4-2210-436c-be84-b941291a268f" containerName="payload-processing" containerID="cri-o://25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7" gracePeriod=25 Oct 30 13:12:33.985750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.985715 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" Oct 30 13:12:33.985880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.985713 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8" event={"ID":"a72af4e9-33c9-4d64-ac6a-821be772a350","Type":"ContainerDied","Data":"c8fd57df97d8bbd03f37395e42c99e26d2dfb2c2f3e939b142e2cc1620d839e4"} Oct 30 13:12:33.985880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:33.985859 5693 scope.go:122] "RemoveContainer" containerID="19e174aea08d52139df15698c9e17c3623ae30921ecede19d7706479e482b079" Oct 30 13:12:34.059204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.059132 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8"] Oct 30 13:12:34.071351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.071328 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d6a10701-78f6977c4-wkgn8"] Oct 30 13:12:34.223743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.223721 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:34.252730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.252696 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72af4e9-33c9-4d64-ac6a-821be772a350" path="/var/lib/kubelet/pods/a72af4e9-33c9-4d64-ac6a-821be772a350/volumes" Oct 30 13:12:34.332133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.332047 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6\" (UniqueName: \"kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6\") pod \"8149c5e4-2210-436c-be84-b941291a268f\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " Oct 30 13:12:34.332133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.332086 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume\") pod \"8149c5e4-2210-436c-be84-b941291a268f\" (UID: \"8149c5e4-2210-436c-be84-b941291a268f\") " Oct 30 13:12:34.332689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.332566 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume" pod "8149c5e4-2210-436c-be84-b941291a268f" (UID: "8149c5e4-2210-436c-be84-b941291a268f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:34.334257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.334227 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6" pod "8149c5e4-2210-436c-be84-b941291a268f" (UID: "8149c5e4-2210-436c-be84-b941291a268f"). InnerVolumeSpecName "kube-api-access-82wt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:34.433708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.433675 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wt6\" (UniqueName: \"kubernetes.io/projected/8149c5e4-2210-436c-be84-b941291a268f-kube-api-access-82wt6\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:34.433708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.433703 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8149c5e4-2210-436c-be84-b941291a268f-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:34.644673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.644570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:34.994241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.994196 5693 generic.go:356] "Generic (PLEG): container finished" podID="8149c5e4-2210-436c-be84-b941291a268f" containerID="25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7" exitCode=0 Oct 30 13:12:34.994241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.994240 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" Oct 30 13:12:34.994473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.994265 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" event={"ID":"8149c5e4-2210-436c-be84-b941291a268f","Type":"ContainerDied","Data":"25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7"} Oct 30 13:12:34.994473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.994321 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b" event={"ID":"8149c5e4-2210-436c-be84-b941291a268f","Type":"ContainerDied","Data":"a3d69ee5a0bddf840a96b03338f5aa2c585f2f97b8d4997234ac8ab0dc7c9320"} Oct 30 13:12:34.994473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:34.994347 5693 scope.go:122] "RemoveContainer" containerID="25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7" Oct 30 13:12:35.003764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.003742 5693 scope.go:122] "RemoveContainer" containerID="25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7" Oct 30 13:12:35.004066 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:35.004045 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7\": container with ID starting with 25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7 not found: ID does not exist" containerID="25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7" Oct 30 13:12:35.004159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.004072 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7"} err="failed to get container status \"25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7\": rpc error: code = NotFound desc = could not find container \"25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7\": container with ID starting with 25108f4ae03a97ce00f7b7c85c5a7e020f750c5442203b336986eae83789dfe7 not found: ID does not exist" Oct 30 13:12:35.087856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.087824 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b"] Oct 30 13:12:35.102170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.102140 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc4197ef-5c6f9458fb-cwc5b"] Oct 30 13:12:35.123287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.123255 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f"] Oct 30 13:12:35.123533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.123507 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" podUID="68896e1d-68b8-42be-b3c3-19b4c9ae166e" containerName="maas-api" containerID="cri-o://55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa" gracePeriod=30 Oct 30 13:12:35.156738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.156706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.300057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.300021 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.440035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.440005 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:35.544086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.543963 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs\") pod \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " Oct 30 13:12:35.544086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.544026 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca\") pod \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " Oct 30 13:12:35.544086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.544055 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\") pod \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " Oct 30 13:12:35.544373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.544125 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr\" (UniqueName: \"kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr\") pod \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\" (UID: \"68896e1d-68b8-42be-b3c3-19b4c9ae166e\") " Oct 30 13:12:35.544747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.544715 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca" pod "68896e1d-68b8-42be-b3c3-19b4c9ae166e" (UID: "68896e1d-68b8-42be-b3c3-19b4c9ae166e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:35.547558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.547527 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr" pod "68896e1d-68b8-42be-b3c3-19b4c9ae166e" (UID: "68896e1d-68b8-42be-b3c3-19b4c9ae166e"). InnerVolumeSpecName "kube-api-access-hc7pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:35.550462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.550432 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs" pod "68896e1d-68b8-42be-b3c3-19b4c9ae166e" (UID: "68896e1d-68b8-42be-b3c3-19b4c9ae166e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:35.551473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.551446 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls" pod "68896e1d-68b8-42be-b3c3-19b4c9ae166e" (UID: "68896e1d-68b8-42be-b3c3-19b4c9ae166e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:35.646357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.646318 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.646357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.646360 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68896e1d-68b8-42be-b3c3-19b4c9ae166e-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.646630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.646376 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68896e1d-68b8-42be-b3c3-19b4c9ae166e-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.646630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.646391 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7pr\" (UniqueName: \"kubernetes.io/projected/68896e1d-68b8-42be-b3c3-19b4c9ae166e-kube-api-access-hc7pr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.882930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:35.882837 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.028755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.028714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.032890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.032857 5693 generic.go:356] "Generic (PLEG): container finished" podID="68896e1d-68b8-42be-b3c3-19b4c9ae166e" containerID="55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa" exitCode=0 Oct 30 13:12:36.033047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.032885 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" event={"ID":"68896e1d-68b8-42be-b3c3-19b4c9ae166e","Type":"ContainerDied","Data":"55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa"} Oct 30 13:12:36.033047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.032931 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" Oct 30 13:12:36.033047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.032987 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f" event={"ID":"68896e1d-68b8-42be-b3c3-19b4c9ae166e","Type":"ContainerDied","Data":"001a2f7a3edd604b8bd89d3b1c5e5e81a9a71fc1fa74be4b35d79bb8cfd3a073"} Oct 30 13:12:36.033047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.033021 5693 scope.go:122] "RemoveContainer" containerID="55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa" Oct 30 13:12:36.044898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.044878 5693 scope.go:122] "RemoveContainer" containerID="55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa" Oct 30 13:12:36.045406 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:36.045355 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa\": container with ID starting with 55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa not found: ID does not exist" containerID="55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa" Oct 30 13:12:36.045406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.045394 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa"} err="failed to get container status \"55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa\": rpc error: code = NotFound desc = could not find container \"55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa\": container with ID starting with 55e8e36acb40c03d22dd752a8ee7aaf63c83986879ab6a41bbe65db84a4fd7aa not found: ID does not exist" Oct 30 13:12:36.085516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.085485 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f"] Oct 30 13:12:36.097196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.097156 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc4197ef-5bff877fcb-s4h8f"] Oct 30 13:12:36.172107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.172071 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q"] Oct 30 13:12:36.172979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.172339 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" containerID="cri-o://a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6" gracePeriod=30 Oct 30 13:12:36.174590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.174567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.253930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.253901 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68896e1d-68b8-42be-b3c3-19b4c9ae166e" path="/var/lib/kubelet/pods/68896e1d-68b8-42be-b3c3-19b4c9ae166e/volumes" Oct 30 13:12:36.254257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.254243 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8149c5e4-2210-436c-be84-b941291a268f" path="/var/lib/kubelet/pods/8149c5e4-2210-436c-be84-b941291a268f/volumes" Oct 30 13:12:36.323689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.323658 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.501880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.501847 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh"] Oct 30 13:12:36.674652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.674615 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.813299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:36.813224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:37.093362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.092727 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:37.520397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520366 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57"] Oct 30 13:12:37.520921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520877 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1eaaf2c9-2166-428a-8607-3b5caa50f76e" containerName="payload-processing" Oct 30 13:12:37.520921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520890 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eaaf2c9-2166-428a-8607-3b5caa50f76e" containerName="payload-processing" Oct 30 13:12:37.520921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520899 5693 cpu_manager.go:414] "containerMap: removing container" podUID="68896e1d-68b8-42be-b3c3-19b4c9ae166e" containerName="maas-api" Oct 30 13:12:37.520921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520904 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68896e1d-68b8-42be-b3c3-19b4c9ae166e" containerName="maas-api" Oct 30 13:12:37.520921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520923 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8149c5e4-2210-436c-be84-b941291a268f" containerName="payload-processing" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520929 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8149c5e4-2210-436c-be84-b941291a268f" containerName="payload-processing" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520936 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a72af4e9-33c9-4d64-ac6a-821be772a350" containerName="maas-api" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.520941 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72af4e9-33c9-4d64-ac6a-821be772a350" containerName="maas-api" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.521015 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8149c5e4-2210-436c-be84-b941291a268f" containerName="payload-processing" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.521022 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72af4e9-33c9-4d64-ac6a-821be772a350" containerName="maas-api" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.521034 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eaaf2c9-2166-428a-8607-3b5caa50f76e" containerName="payload-processing" Oct 30 13:12:37.521188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.521039 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="68896e1d-68b8-42be-b3c3-19b4c9ae166e" containerName="maas-api" Oct 30 13:12:37.524867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.524845 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.551337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.551308 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57"] Oct 30 13:12:37.560206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.560182 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ec41e5-openshift-default-dockercfg-6fkcb\"" Oct 30 13:12:37.570718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.570689 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzxt\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.570855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.570750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.570924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.570855 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.570971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.570924 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.571006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.570992 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.571044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.571034 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.571084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.571059 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.571214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.571187 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.571285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.571269 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.628497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.628445 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673303 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673469 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673531 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzxt\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673677 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673718 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673784 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.673927 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.674008 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.676263 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.676507 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.676765 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.677161 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.681621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.678615 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.688158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.688122 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.705038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.704993 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.706522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.705682 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.715244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.715213 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzxt\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:37.799181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.799081 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:37.840799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:37.840768 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:38.065585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:38.065489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:38.159729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:38.159703 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57"] Oct 30 13:12:38.162379 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:38.162349 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984445f4_1af1_460d_a436_7d55601d2866.slice/crio-defc330e5184b09e25d801e5c6866661c0348a65909da254fdae328d438d35e5 WatchSource:0}: Error finding container defc330e5184b09e25d801e5c6866661c0348a65909da254fdae328d438d35e5: Status 404 returned error can't find the container with id defc330e5184b09e25d801e5c6866661c0348a65909da254fdae328d438d35e5 Oct 30 13:12:38.165054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:38.165023 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:38.165121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:38.165085 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:38.165121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:38.165114 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:39.052715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.052679 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" event={"ID":"984445f4-1af1-460d-a436-7d55601d2866","Type":"ContainerStarted","Data":"854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481"} Oct 30 13:12:39.053118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.052733 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" event={"ID":"984445f4-1af1-460d-a436-7d55601d2866","Type":"ContainerStarted","Data":"defc330e5184b09e25d801e5c6866661c0348a65909da254fdae328d438d35e5"} Oct 30 13:12:39.109549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.109499 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podStartSLOduration=2.109484215 podStartE2EDuration="2.109484215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:39.108109305 +0000 UTC m=+534.974727477" watchObservedRunningTime="2026-10-30 13:12:39.109484215 +0000 UTC m=+534.976102389" Oct 30 13:12:39.120538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.120466 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.283621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.280505 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.452286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.452197 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.483233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.483200 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7"] Oct 30 13:12:39.483447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.483411 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" podUID="1210f0cc-9e7d-4eda-b076-cd73b0470d4b" containerName="maas-api" containerID="cri-o://62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878" gracePeriod=30 Oct 30 13:12:39.600757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.600725 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.751440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.751415 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:39.805925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.805895 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca\") pod \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " Oct 30 13:12:39.806117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.806034 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs\") pod \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " Oct 30 13:12:39.806117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.806095 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh\" (UniqueName: \"kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh\") pod \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " Oct 30 13:12:39.806242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.806164 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls\") pod \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\" (UID: \"1210f0cc-9e7d-4eda-b076-cd73b0470d4b\") " Oct 30 13:12:39.806315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.806294 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca" pod "1210f0cc-9e7d-4eda-b076-cd73b0470d4b" (UID: "1210f0cc-9e7d-4eda-b076-cd73b0470d4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:39.806650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.806625 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.808299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.808267 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs" pod "1210f0cc-9e7d-4eda-b076-cd73b0470d4b" (UID: "1210f0cc-9e7d-4eda-b076-cd73b0470d4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:39.808447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.808426 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh" pod "1210f0cc-9e7d-4eda-b076-cd73b0470d4b" (UID: "1210f0cc-9e7d-4eda-b076-cd73b0470d4b"). InnerVolumeSpecName "kube-api-access-4w6kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:39.809616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.809577 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls" pod "1210f0cc-9e7d-4eda-b076-cd73b0470d4b" (UID: "1210f0cc-9e7d-4eda-b076-cd73b0470d4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:39.840992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.840966 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:39.846189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.846164 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:39.908426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.908394 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.908426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.908428 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.908635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.908443 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6kh\" (UniqueName: \"kubernetes.io/projected/1210f0cc-9e7d-4eda-b076-cd73b0470d4b-kube-api-access-4w6kh\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.989071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:39.988980 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:40.059148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.059109 5693 generic.go:356] "Generic (PLEG): container finished" podID="1210f0cc-9e7d-4eda-b076-cd73b0470d4b" containerID="62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878" exitCode=0 Oct 30 13:12:40.059587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.059169 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" Oct 30 13:12:40.059587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.059181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" event={"ID":"1210f0cc-9e7d-4eda-b076-cd73b0470d4b","Type":"ContainerDied","Data":"62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878"} Oct 30 13:12:40.059587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.059241 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7" event={"ID":"1210f0cc-9e7d-4eda-b076-cd73b0470d4b","Type":"ContainerDied","Data":"0fb0adcdb193fe16222675710e648f568d2738d9a273f698ab0e2623486b3969"} Oct 30 13:12:40.059587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.059272 5693 scope.go:122] "RemoveContainer" containerID="62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878" Oct 30 13:12:40.059841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.059725 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:40.060776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.060758 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:12:40.073687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.073592 5693 scope.go:122] "RemoveContainer" containerID="62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878" Oct 30 13:12:40.074021 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:40.074000 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878\": container with ID starting with 62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878 not found: ID does not exist" containerID="62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878" Oct 30 13:12:40.074115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.074028 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878"} err="failed to get container status \"62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878\": rpc error: code = NotFound desc = could not find container \"62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878\": container with ID starting with 62afdcb85b5e004ae236986ec101f3f64a7ba295c3ba16d3539234403de4a878 not found: ID does not exist" Oct 30 13:12:40.143550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.143460 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:40.154871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.154832 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7"] Oct 30 13:12:40.165519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.165491 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-deb5d83f-56488d54fb-r5gt7"] Oct 30 13:12:40.254200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.254131 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1210f0cc-9e7d-4eda-b076-cd73b0470d4b" path="/var/lib/kubelet/pods/1210f0cc-9e7d-4eda-b076-cd73b0470d4b/volumes" Oct 30 13:12:40.297089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.297057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:40.657543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.657455 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:40.854298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.854266 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4"] Oct 30 13:12:40.854561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.854522 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" containerID="cri-o://a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f" gracePeriod=30 Oct 30 13:12:40.913250 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:40.913164 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.061164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.061129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.438902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.438842 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:12:41.503102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.503049 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" podUID="327aa54a-867d-4c9e-ab72-315010ccaa52" containerName="payload-processing" containerID="cri-o://66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5" gracePeriod=25 Oct 30 13:12:41.527814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.527790 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.527940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.527840 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.527940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.527895 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.527993 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528046 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528055 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:41.528151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528128 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528161 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528215 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:41.528349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528263 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528261 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:41.528349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528302 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data\") pod \"1d865eba-0229-4aa7-aecf-8a251371ef74\" (UID: \"1d865eba-0229-4aa7-aecf-8a251371ef74\") " Oct 30 13:12:41.528837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528615 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:41.528837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528724 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:41.528837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528769 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.528837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528790 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.528837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528804 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d865eba-0229-4aa7-aecf-8a251371ef74-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.528837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.528817 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.530770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.530734 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "kube-api-access-wdnv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:41.531074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.531052 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:41.531074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.531062 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:41.531189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.531081 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token" pod "1d865eba-0229-4aa7-aecf-8a251371ef74" (UID: "1d865eba-0229-4aa7-aecf-8a251371ef74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:41.630379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.630338 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.630379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.630373 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.630573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.630387 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnv9\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-kube-api-access-wdnv9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.630573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.630397 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.630573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.630405 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d865eba-0229-4aa7-aecf-8a251371ef74-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.727782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.727744 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.743950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.743927 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:41.832009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.831974 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf\" (UniqueName: \"kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf\") pod \"327aa54a-867d-4c9e-ab72-315010ccaa52\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " Oct 30 13:12:41.832174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.832094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume\") pod \"327aa54a-867d-4c9e-ab72-315010ccaa52\" (UID: \"327aa54a-867d-4c9e-ab72-315010ccaa52\") " Oct 30 13:12:41.832452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.832430 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume" pod "327aa54a-867d-4c9e-ab72-315010ccaa52" (UID: "327aa54a-867d-4c9e-ab72-315010ccaa52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:41.834129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.834104 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf" pod "327aa54a-867d-4c9e-ab72-315010ccaa52" (UID: "327aa54a-867d-4c9e-ab72-315010ccaa52"). InnerVolumeSpecName "kube-api-access-bnwhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:41.933468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.933435 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327aa54a-867d-4c9e-ab72-315010ccaa52-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.933468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.933462 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnwhf\" (UniqueName: \"kubernetes.io/projected/327aa54a-867d-4c9e-ab72-315010ccaa52-kube-api-access-bnwhf\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.978886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:41.978803 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.074098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.074059 5693 generic.go:356] "Generic (PLEG): container finished" podID="327aa54a-867d-4c9e-ab72-315010ccaa52" containerID="66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5" exitCode=0 Oct 30 13:12:42.074634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.074112 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" Oct 30 13:12:42.074634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.074136 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" event={"ID":"327aa54a-867d-4c9e-ab72-315010ccaa52","Type":"ContainerDied","Data":"66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5"} Oct 30 13:12:42.074634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.074193 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh" event={"ID":"327aa54a-867d-4c9e-ab72-315010ccaa52","Type":"ContainerDied","Data":"495e1bbf7971d785f08452dfcc53852484e92cdfa84e2b9b8d363a98bb358ab9"} Oct 30 13:12:42.074634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.074222 5693 scope.go:122] "RemoveContainer" containerID="66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5" Oct 30 13:12:42.076357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.076322 5693 generic.go:356] "Generic (PLEG): container finished" podID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerID="a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6" exitCode=0 Oct 30 13:12:42.076477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.076375 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" Oct 30 13:12:42.076477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.076374 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" event={"ID":"1d865eba-0229-4aa7-aecf-8a251371ef74","Type":"ContainerDied","Data":"a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6"} Oct 30 13:12:42.076477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.076446 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q" event={"ID":"1d865eba-0229-4aa7-aecf-8a251371ef74","Type":"ContainerDied","Data":"3109b743d836552decffd70f5673e0554b29ed36024831ab8b23fee6a2a7fbec"} Oct 30 13:12:42.089538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.089510 5693 scope.go:122] "RemoveContainer" containerID="66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5" Oct 30 13:12:42.090039 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:42.089908 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5\": container with ID starting with 66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5 not found: ID does not exist" containerID="66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5" Oct 30 13:12:42.090039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.089945 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5"} err="failed to get container status \"66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5\": rpc error: code = NotFound desc = could not find container \"66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5\": container with ID starting with 66af9bf570694614d8bb0f0075262e825cea64c73c20dc58d0c41194fb1593d5 not found: ID does not exist" Oct 30 13:12:42.090039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.089966 5693 scope.go:122] "RemoveContainer" containerID="a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6" Oct 30 13:12:42.098978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.098961 5693 scope.go:122] "RemoveContainer" containerID="f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a" Oct 30 13:12:42.107947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.107928 5693 scope.go:122] "RemoveContainer" containerID="a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6" Oct 30 13:12:42.108714 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:42.108686 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6\": container with ID starting with a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6 not found: ID does not exist" containerID="a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6" Oct 30 13:12:42.108869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.108720 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6"} err="failed to get container status \"a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6\": rpc error: code = NotFound desc = could not find container \"a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6\": container with ID starting with a8756bfdd9bafe5c720e575159915a0ff527b219f5eba2443619c22f8012baa6 not found: ID does not exist" Oct 30 13:12:42.108869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.108745 5693 scope.go:122] "RemoveContainer" containerID="f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a" Oct 30 13:12:42.109092 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:42.109068 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a\": container with ID starting with f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a not found: ID does not exist" containerID="f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a" Oct 30 13:12:42.109155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.109097 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a"} err="failed to get container status \"f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a\": rpc error: code = NotFound desc = could not find container \"f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a\": container with ID starting with f7b73202018781db24127859818a095a3f60d87597aed65511bb583c9e40af2a not found: ID does not exist" Oct 30 13:12:42.112528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.112508 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4"] Oct 30 13:12:42.113204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113183 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" Oct 30 13:12:42.113204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113203 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" Oct 30 13:12:42.113334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113228 5693 cpu_manager.go:414] "containerMap: removing container" podUID="327aa54a-867d-4c9e-ab72-315010ccaa52" containerName="payload-processing" Oct 30 13:12:42.113334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113233 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327aa54a-867d-4c9e-ab72-315010ccaa52" containerName="payload-processing" Oct 30 13:12:42.113334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113254 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1210f0cc-9e7d-4eda-b076-cd73b0470d4b" containerName="maas-api" Oct 30 13:12:42.113334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113263 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1210f0cc-9e7d-4eda-b076-cd73b0470d4b" containerName="maas-api" Oct 30 13:12:42.113452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113336 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="327aa54a-867d-4c9e-ab72-315010ccaa52" containerName="payload-processing" Oct 30 13:12:42.113452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113348 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" Oct 30 13:12:42.113452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113354 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1210f0cc-9e7d-4eda-b076-cd73b0470d4b" containerName="maas-api" Oct 30 13:12:42.113452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.113362 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" Oct 30 13:12:42.118109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.118091 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.139513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.139478 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4"] Oct 30 13:12:42.141469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.141449 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh"] Oct 30 13:12:42.141579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.141539 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f9ebb5-openshift-default-dockercfg-259p4\"" Oct 30 13:12:42.146533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.146511 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.147722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.147699 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-deb5d83f-57cdbfc44-lwmlh"] Oct 30 13:12:42.182767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.182735 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q"] Oct 30 13:12:42.195078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.195050 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fc4197ef-openshift-default-66d46c94c5-pz47q"] Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268423 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268471 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268504 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268560 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268686 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665kw\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268725 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268826 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.271679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.268851 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.299623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.299065 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.301955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.301867 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" path="/var/lib/kubelet/pods/1d865eba-0229-4aa7-aecf-8a251371ef74/volumes" Oct 30 13:12:42.304026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.304002 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327aa54a-867d-4c9e-ab72-315010ccaa52" path="/var/lib/kubelet/pods/327aa54a-867d-4c9e-ab72-315010ccaa52/volumes" Oct 30 13:12:42.370179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370145 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370184 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370205 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370259 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370360 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665kw\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370396 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370434 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370500 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370526 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370690 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.370867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370726 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.371021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370995 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.371134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.370991 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.371435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.371418 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.372900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.372881 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.373041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.373020 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.389248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.389212 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.389811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.389793 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-665kw\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.442885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.442850 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:42.448548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.448511 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.652681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.652569 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.756658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.756627 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4"] Oct 30 13:12:42.760004 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:42.759979 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b856a0_3adb_4403_a497_91b12ad5d85d.slice/crio-6470f7483344d0e0a1b81fd7a26f38c8929e852e8e57ff5b8e60ad31964ee056 WatchSource:0}: Error finding container 6470f7483344d0e0a1b81fd7a26f38c8929e852e8e57ff5b8e60ad31964ee056: Status 404 returned error can't find the container with id 6470f7483344d0e0a1b81fd7a26f38c8929e852e8e57ff5b8e60ad31964ee056 Oct 30 13:12:42.762324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.762293 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:42.762389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.762356 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:42.762389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.762384 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:42.789263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.789234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.935823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:42.935571 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:43.085351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.085312 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" event={"ID":"b8b856a0-3adb-4403-a497-91b12ad5d85d","Type":"ContainerStarted","Data":"2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92"} Oct 30 13:12:43.085804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.085381 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" event={"ID":"b8b856a0-3adb-4403-a497-91b12ad5d85d","Type":"ContainerStarted","Data":"6470f7483344d0e0a1b81fd7a26f38c8929e852e8e57ff5b8e60ad31964ee056"} Oct 30 13:12:43.127420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.127358 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podStartSLOduration=1.127343041 podStartE2EDuration="1.127343041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:43.125103807 +0000 UTC m=+538.991721971" watchObservedRunningTime="2026-10-30 13:12:43.127343041 +0000 UTC m=+538.993961213" Oct 30 13:12:43.444079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.444048 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:43.449078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.449054 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:43.732443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.732409 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:43.807747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.807717 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b"] Oct 30 13:12:43.808252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.808238 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" Oct 30 13:12:43.808302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.808254 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d865eba-0229-4aa7-aecf-8a251371ef74" containerName="istio-proxy" Oct 30 13:12:43.811313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.811293 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:43.836051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.836005 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b"] Oct 30 13:12:43.857810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.857787 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ec41e5-dockercfg-kgqml\"" Oct 30 13:12:43.888843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.888815 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv6h\" (UniqueName: \"kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h\") pod \"payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:43.888970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.888933 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:43.894529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.894498 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:43.899175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.899130 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ec41e5\"" Oct 30 13:12:43.990090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.989997 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv6h\" (UniqueName: \"kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h\") pod \"payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:43.990251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.990134 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:43.990863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:43.990837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:44.009178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.009150 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv6h\" (UniqueName: \"kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h\") pod \"payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:44.092653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.092621 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:44.093735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.093708 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:12:44.139128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.139071 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:44.326861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.326838 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:44.458955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.458926 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b"] Oct 30 13:12:44.505994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:44.505958 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.099507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:45.099464 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" event={"ID":"16d031a8-4853-490f-baf9-0e39d19ad754","Type":"ContainerStarted","Data":"71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1"} Oct 30 13:12:45.100033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:45.099529 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" event={"ID":"16d031a8-4853-490f-baf9-0e39d19ad754","Type":"ContainerStarted","Data":"38d86f78ea8c80819ee3f84347972cc7b35b3547c0ca8a478d8460c4f3cbbeaa"} Oct 30 13:12:45.138145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:45.138093 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" podStartSLOduration=2.138077401 podStartE2EDuration="2.138077401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:45.137083641 +0000 UTC m=+541.003701814" watchObservedRunningTime="2026-10-30 13:12:45.138077401 +0000 UTC m=+541.004695574" Oct 30 13:12:45.605826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:45.605791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.759949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:45.759920 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.107407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.107381 5693 generic.go:356] "Generic (PLEG): container finished" podID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerID="a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f" exitCode=0 Oct 30 13:12:46.107750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.107463 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" event={"ID":"32655306-fd82-4b28-aff7-4ce4a124e57e","Type":"ContainerDied","Data":"a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f"} Oct 30 13:12:46.107750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.107525 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" event={"ID":"32655306-fd82-4b28-aff7-4ce4a124e57e","Type":"ContainerDied","Data":"056435dae9f55419571a4716b170cb9648e9728aecb157ab03e96d70a6056eae"} Oct 30 13:12:46.107750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.107549 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="056435dae9f55419571a4716b170cb9648e9728aecb157ab03e96d70a6056eae" Oct 30 13:12:46.107750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.107565 5693 scope.go:122] "RemoveContainer" containerID="0d507e440f21ba4ec4f916eff993aef966ff6285e1db721b093721fa7402e490" Oct 30 13:12:46.107934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.107836 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:46.114536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.114514 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:12:46.134283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134225 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134275 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134332 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134366 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134403 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134455 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134485 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134529 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134565 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert\") pod \"32655306-fd82-4b28-aff7-4ce4a124e57e\" (UID: \"32655306-fd82-4b28-aff7-4ce4a124e57e\") " Oct 30 13:12:46.134740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134582 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:46.134740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134676 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:46.134902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134811 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:46.135017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.134992 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:46.135102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.135038 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:46.135539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.135169 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.135539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.135194 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.135539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.135208 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.135539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.135223 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32655306-fd82-4b28-aff7-4ce4a124e57e-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.135539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.135237 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.136581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.136558 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:46.136736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.136715 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "kube-api-access-pzpsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:46.136823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.136802 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:46.137344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.137314 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo" pod "32655306-fd82-4b28-aff7-4ce4a124e57e" (UID: "32655306-fd82-4b28-aff7-4ce4a124e57e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:46.236831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.236794 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.236831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.236822 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.236831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.236831 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzpsh\" (UniqueName: \"kubernetes.io/projected/32655306-fd82-4b28-aff7-4ce4a124e57e-kube-api-access-pzpsh\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.236831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.236840 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32655306-fd82-4b28-aff7-4ce4a124e57e-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.510063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.510017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.647211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.647175 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.788631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:46.788530 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.114809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.114731 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4" Oct 30 13:12:47.219488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.219448 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4"] Oct 30 13:12:47.221399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.221371 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-deb5d83f-openshift-default-75f857cc79-9ghw4"] Oct 30 13:12:47.635246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.635213 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.690461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.690432 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k"] Oct 30 13:12:47.691205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.691179 5693 cpu_manager.go:414] "containerMap: removing container" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" Oct 30 13:12:47.691402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.691363 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" Oct 30 13:12:47.691506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.691407 5693 cpu_manager.go:414] "containerMap: removing container" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" Oct 30 13:12:47.691506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.691416 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" Oct 30 13:12:47.691591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.691581 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" Oct 30 13:12:47.691650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.691611 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" containerName="istio-proxy" Oct 30 13:12:47.696132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.696114 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk"] Oct 30 13:12:47.696251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.696233 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.700079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.700058 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:47.713530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.713509 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k"] Oct 30 13:12:47.714263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.714237 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f9ebb5\"" Oct 30 13:12:47.716253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.716231 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk"] Oct 30 13:12:47.754692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.754661 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlb9\" (UniqueName: \"kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9\") pod \"payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:47.754866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.754712 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:47.754866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.754842 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.755002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.754893 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5x9\" (UniqueName: \"kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9\") pod \"payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.780376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.780337 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.855783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.855663 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.855783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.855749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5x9\" (UniqueName: \"kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9\") pod \"payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.856758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.855863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlb9\" (UniqueName: \"kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9\") pod \"payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:47.856758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.855911 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:47.869494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.858107 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:47.870648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.870362 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.906667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.903409 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5x9\" (UniqueName: \"kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9\") pod \"payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:47.906667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:47.904072 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlb9\" (UniqueName: \"kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9\") pod \"payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:48.024622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.017819 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:48.024622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.020393 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:48.252810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.252777 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32655306-fd82-4b28-aff7-4ce4a124e57e" path="/var/lib/kubelet/pods/32655306-fd82-4b28-aff7-4ce4a124e57e/volumes" Oct 30 13:12:48.315629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.312152 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z"] Oct 30 13:12:48.315629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.312397 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" containerID="cri-o://4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc" gracePeriod=30 Oct 30 13:12:48.365113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.365080 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k"] Oct 30 13:12:48.369973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:48.369940 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41b1373_0002_4a40_9a9d_abff679404c0.slice/crio-c642fe12b440de70f8c71fdbe57458938406f3280e9c5465301acc6acc94ba1e WatchSource:0}: Error finding container c642fe12b440de70f8c71fdbe57458938406f3280e9c5465301acc6acc94ba1e: Status 404 returned error can't find the container with id c642fe12b440de70f8c71fdbe57458938406f3280e9c5465301acc6acc94ba1e Oct 30 13:12:48.379803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.379782 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk"] Oct 30 13:12:48.382973 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:48.382948 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201a039d_07be_4f7a_b194_7a536c6fdfcc.slice/crio-9c63761b9f6aa65cc53816727b1b8e1304a15e1f4bff5a906e4b7c924d3c3f23 WatchSource:0}: Error finding container 9c63761b9f6aa65cc53816727b1b8e1304a15e1f4bff5a906e4b7c924d3c3f23: Status 404 returned error can't find the container with id 9c63761b9f6aa65cc53816727b1b8e1304a15e1f4bff5a906e4b7c924d3c3f23 Oct 30 13:12:48.404259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.404218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:48.633523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:48.633264 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:49.142867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.142826 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" event={"ID":"a41b1373-0002-4a40-9a9d-abff679404c0","Type":"ContainerStarted","Data":"dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e"} Oct 30 13:12:49.143053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.143034 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:12:49.143142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.143090 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" event={"ID":"a41b1373-0002-4a40-9a9d-abff679404c0","Type":"ContainerStarted","Data":"c642fe12b440de70f8c71fdbe57458938406f3280e9c5465301acc6acc94ba1e"} Oct 30 13:12:49.144537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.144504 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" event={"ID":"201a039d-07be-4f7a-b194-7a536c6fdfcc","Type":"ContainerStarted","Data":"3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4"} Oct 30 13:12:49.144664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.144546 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" event={"ID":"201a039d-07be-4f7a-b194-7a536c6fdfcc","Type":"ContainerStarted","Data":"9c63761b9f6aa65cc53816727b1b8e1304a15e1f4bff5a906e4b7c924d3c3f23"} Oct 30 13:12:49.144664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.144571 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:12:49.183944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.183900 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" podStartSLOduration=2.183888168 podStartE2EDuration="2.183888168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:49.183581331 +0000 UTC m=+545.050199503" watchObservedRunningTime="2026-10-30 13:12:49.183888168 +0000 UTC m=+545.050506341" Oct 30 13:12:49.212645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.212617 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:49.221376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.221335 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" podStartSLOduration=2.22132256 podStartE2EDuration="2.22132256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:49.219708488 +0000 UTC m=+545.086326661" watchObservedRunningTime="2026-10-30 13:12:49.22132256 +0000 UTC m=+545.087940730" Oct 30 13:12:49.720280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.720249 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:49.868179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:49.868139 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:50.108280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.108245 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4"] Oct 30 13:12:50.109859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.109789 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" podUID="6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" containerName="maas-api" containerID="cri-o://9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca" gracePeriod=30 Oct 30 13:12:50.207384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.207340 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:50.386267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.386242 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:50.396238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.396215 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs\") pod \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " Oct 30 13:12:50.396367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.396254 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs\" (UniqueName: \"kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs\") pod \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " Oct 30 13:12:50.396367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.396299 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca\") pod \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " Oct 30 13:12:50.396453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.396395 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\") pod \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\" (UID: \"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0\") " Oct 30 13:12:50.396797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.396765 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca" pod "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" (UID: "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:50.398214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.398191 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs" pod "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" (UID: "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:50.398317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.398302 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls" pod "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" (UID: "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:50.398457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.398436 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs" pod "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" (UID: "6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0"). InnerVolumeSpecName "kube-api-access-544rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:50.497840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.497807 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:50.497840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.497840 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:50.498030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.497856 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544rs\" (UniqueName: \"kubernetes.io/projected/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-kube-api-access-544rs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:50.498030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.497871 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:50.664656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.664562 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d"] Oct 30 13:12:50.665127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.665108 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" containerName="maas-api" Oct 30 13:12:50.665180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.665130 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" containerName="maas-api" Oct 30 13:12:50.665236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.665226 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" containerName="maas-api" Oct 30 13:12:50.668718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.668703 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.684809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.684784 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f9ebb5-serving-cert\"" Oct 30 13:12:50.686952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.686844 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d"] Oct 30 13:12:50.699734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.699705 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.699859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.699750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwxg\" (UniqueName: \"kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.699859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.699828 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.699991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.699969 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.702929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.702887 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:50.801280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.801246 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.801767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.801288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwxg\" (UniqueName: \"kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.801767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.801305 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.801767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.801371 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.802234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.802208 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.803952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.803930 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.804037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.803954 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.820171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.820147 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwxg\" (UniqueName: \"kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg\") pod \"maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.852297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.852268 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:50.982730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.982701 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:50.988338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:50.988313 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.126618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.122879 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.161652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.161587 5693 generic.go:356] "Generic (PLEG): container finished" podID="6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" containerID="9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca" exitCode=0 Oct 30 13:12:51.161652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.161650 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" Oct 30 13:12:51.161835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.161632 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" event={"ID":"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0","Type":"ContainerDied","Data":"9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca"} Oct 30 13:12:51.161835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.161756 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4" event={"ID":"6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0","Type":"ContainerDied","Data":"cac0a14abdf9bbc61db90a060fb171248532378c9c05f794fceb32e789338b72"} Oct 30 13:12:51.161835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.161787 5693 scope.go:122] "RemoveContainer" containerID="9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca" Oct 30 13:12:51.171498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.171479 5693 scope.go:122] "RemoveContainer" containerID="9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca" Oct 30 13:12:51.171830 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:51.171806 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca\": container with ID starting with 9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca not found: ID does not exist" containerID="9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca" Oct 30 13:12:51.171930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.171835 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca"} err="failed to get container status \"9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca\": rpc error: code = NotFound desc = could not find container \"9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca\": container with ID starting with 9095f2d7cb97a6a3b848053a14702b0085b46ffae906bd62b71441e3c5a498ca not found: ID does not exist" Oct 30 13:12:51.210433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.210400 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4"] Oct 30 13:12:51.213897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.213875 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb3b54cb-fb4b5cf5-p4bm4"] Oct 30 13:12:51.283699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.283627 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d"] Oct 30 13:12:51.286960 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:51.286931 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3e16f5_7af6_428a_a80e_fc16c17b6f51.slice/crio-bf53e50e2f656ddd71214572092d92202a2184adac595b2f28c4f200c2da3028 WatchSource:0}: Error finding container bf53e50e2f656ddd71214572092d92202a2184adac595b2f28c4f200c2da3028: Status 404 returned error can't find the container with id bf53e50e2f656ddd71214572092d92202a2184adac595b2f28c4f200c2da3028 Oct 30 13:12:51.779116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.779073 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.908900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:51.908814 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.169753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.169641 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" event={"ID":"1a3e16f5-7af6-428a-a80e-fc16c17b6f51","Type":"ContainerStarted","Data":"1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76"} Oct 30 13:12:52.169753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.169712 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" event={"ID":"1a3e16f5-7af6-428a-a80e-fc16c17b6f51","Type":"ContainerStarted","Data":"bf53e50e2f656ddd71214572092d92202a2184adac595b2f28c4f200c2da3028"} Oct 30 13:12:52.169985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.169849 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:52.199642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.199584 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq"] Oct 30 13:12:52.204094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.204071 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.217684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.217637 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" podStartSLOduration=1.9180951529999999 podStartE2EDuration="2.217619557s" totalImagesPullingTime="299.524404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:52.215961665 +0000 UTC m=+548.082579840" watchObservedRunningTime="2026-10-30 13:12:52.217619557 +0000 UTC m=+548.084237727" Oct 30 13:12:52.222582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.222553 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq"] Oct 30 13:12:52.232161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232132 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232218 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232295 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232355 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232377 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232395 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8p4m\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232413 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.232496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.232458 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.255727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.255698 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0" path="/var/lib/kubelet/pods/6a3e55b3-8cae-4d2e-a3b9-be9c9a5882d0/volumes" Oct 30 13:12:52.283011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.282987 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52af49-openshift-default-dockercfg-tbzp6\"" Oct 30 13:12:52.303319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.303290 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.333750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.333714 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.333905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.333808 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.333905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.333853 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.334019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.333904 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.334222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.334200 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.339313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.338586 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.339313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.338765 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.339313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.339175 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.339313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.339235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.339313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.339260 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8p4m\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.339313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.339299 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.341314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.339380 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.341314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.340208 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.341314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.340779 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.352624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.344161 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.383041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.379181 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.383041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.379424 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.400274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.400057 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8p4m\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.521508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.521459 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:52.555003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.554950 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.841939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.841843 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.858865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.858834 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq"] Oct 30 13:12:52.859772 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:52.859747 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b54d4a7_0815_4fa8_958e_67463f66156e.slice/crio-697d607c3eff629019df241663988ba1194cdd3c0b082bb7435350b2c6bb3140 WatchSource:0}: Error finding container 697d607c3eff629019df241663988ba1194cdd3c0b082bb7435350b2c6bb3140: Status 404 returned error can't find the container with id 697d607c3eff629019df241663988ba1194cdd3c0b082bb7435350b2c6bb3140 Oct 30 13:12:52.861984 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.861953 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:52.862045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.862018 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:52.862077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:52.862045 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:53.107629 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:53.107495 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:12:53.190989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.190947 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" event={"ID":"6b54d4a7-0815-4fa8-958e-67463f66156e","Type":"ContainerStarted","Data":"614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4"} Oct 30 13:12:53.191205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.191024 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" event={"ID":"6b54d4a7-0815-4fa8-958e-67463f66156e","Type":"ContainerStarted","Data":"697d607c3eff629019df241663988ba1194cdd3c0b082bb7435350b2c6bb3140"} Oct 30 13:12:53.275975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.275064 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podStartSLOduration=1.2750448730000001 podStartE2EDuration="1.275044873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:53.274817221 +0000 UTC m=+549.141435404" watchObservedRunningTime="2026-10-30 13:12:53.275044873 +0000 UTC m=+549.141663052" Oct 30 13:12:53.372269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.372155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:53.522508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.522475 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:53.534251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.532437 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:53.535053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.534900 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:53.686620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.686573 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:12:53.775764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.775736 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.775907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.775775 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.775907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.775813 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.775907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.775845 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.776036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.775926 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.776036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.775970 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.776121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776046 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:53.776121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776070 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.776223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776122 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.776223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776163 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9\") pod \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\" (UID: \"645c06d9-c5e7-41eb-ab62-88a69dd722d0\") " Oct 30 13:12:53.776330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776233 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:53.776402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776376 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:53.776504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776484 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:53.776567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776533 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:53.776861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776765 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.776861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776793 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.776861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776807 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.776861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776821 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.776861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.776837 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.778323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.778300 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:53.778457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.778422 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:53.778690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.778673 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:53.778811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.778795 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9" pod "645c06d9-c5e7-41eb-ab62-88a69dd722d0" (UID: "645c06d9-c5e7-41eb-ab62-88a69dd722d0"). InnerVolumeSpecName "kube-api-access-8rxl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:53.878319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.878281 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.878319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.878314 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxl9\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-kube-api-access-8rxl9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.878319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.878326 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:53.878550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:53.878335 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645c06d9-c5e7-41eb-ab62-88a69dd722d0-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:12:54.198248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.198211 5693 generic.go:356] "Generic (PLEG): container finished" podID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerID="4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc" exitCode=0 Oct 30 13:12:54.198673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.198279 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" Oct 30 13:12:54.198673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.198295 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" event={"ID":"645c06d9-c5e7-41eb-ab62-88a69dd722d0","Type":"ContainerDied","Data":"4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc"} Oct 30 13:12:54.198673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.198350 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z" event={"ID":"645c06d9-c5e7-41eb-ab62-88a69dd722d0","Type":"ContainerDied","Data":"08d237e8a5cbbf7ddeffe1a97c9fbd714e81273d94c71cbf6d5a5954cf83a0ff"} Oct 30 13:12:54.198673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.198375 5693 scope.go:122] "RemoveContainer" containerID="4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc" Oct 30 13:12:54.198880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.198858 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:54.200183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.200160 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:12:54.209569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.209543 5693 scope.go:122] "RemoveContainer" containerID="ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7" Oct 30 13:12:54.218356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.218336 5693 scope.go:122] "RemoveContainer" containerID="4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc" Oct 30 13:12:54.218647 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:54.218626 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc\": container with ID starting with 4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc not found: ID does not exist" containerID="4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc" Oct 30 13:12:54.218727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.218656 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc"} err="failed to get container status \"4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc\": rpc error: code = NotFound desc = could not find container \"4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc\": container with ID starting with 4dbd50be3921bcc136bee5a0242433374bc6619cbda3b3ed5dda586cc289e8dc not found: ID does not exist" Oct 30 13:12:54.218727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.218673 5693 scope.go:122] "RemoveContainer" containerID="ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7" Oct 30 13:12:54.218970 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:12:54.218952 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7\": container with ID starting with ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7 not found: ID does not exist" containerID="ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7" Oct 30 13:12:54.219032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.218977 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7"} err="failed to get container status \"ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7\": rpc error: code = NotFound desc = could not find container \"ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7\": container with ID starting with ea027dd0c2e638a4adfe64b51ac7dfc4ffb6cdd52f9913cd55255c8b6836d1d7 not found: ID does not exist" Oct 30 13:12:54.299277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.298308 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z"] Oct 30 13:12:54.300996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.300966 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-662ed421-openshift-default-78cbb8bb49-c5b5z"] Oct 30 13:12:54.910451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.910413 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:54.962370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.962284 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5"] Oct 30 13:12:54.962867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.962850 5693 cpu_manager.go:414] "containerMap: removing container" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" Oct 30 13:12:54.962923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.962868 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" Oct 30 13:12:54.962923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.962900 5693 cpu_manager.go:414] "containerMap: removing container" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" Oct 30 13:12:54.962923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.962905 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" Oct 30 13:12:54.963017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.962975 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" Oct 30 13:12:54.967761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.967736 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:54.983880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.983858 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5"] Oct 30 13:12:54.984625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.984572 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9bf8ad\"" Oct 30 13:12:54.990520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.990494 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztnt5\" (UniqueName: \"kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5\") pod \"payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:54.990670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.990543 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:54.993516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.993490 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t"] Oct 30 13:12:54.994399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.994363 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" containerName="istio-proxy" Oct 30 13:12:54.998106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:54.998084 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.020890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.020845 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9bf8ad-dockercfg-48n58\"" Oct 30 13:12:55.023267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.023242 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t"] Oct 30 13:12:55.056675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.056643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:55.091790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.091753 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjn9\" (UniqueName: \"kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9\") pod \"payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.091996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.091945 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.092063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.092025 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztnt5\" (UniqueName: \"kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5\") pod \"payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:55.092174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.092147 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:55.093039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.093014 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:55.170429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.170310 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztnt5\" (UniqueName: \"kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5\") pod \"payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:55.195625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.194583 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjn9\" (UniqueName: \"kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9\") pod \"payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.197179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.197140 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:55.205659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.202721 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.205659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.203669 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.214453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.214427 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjn9\" (UniqueName: \"kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9\") pod \"payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.281005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.280975 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:55.314223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.314193 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:55.603837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.603807 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5"] Oct 30 13:12:55.607450 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:55.607421 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe62a5bc_0e09_4d0e_a514_14b28e9b98df.slice/crio-b2187312e3363f2ee22790c5ed87330f51717dd17e3faf3918e1bb139dcbf2f2 WatchSource:0}: Error finding container b2187312e3363f2ee22790c5ed87330f51717dd17e3faf3918e1bb139dcbf2f2: Status 404 returned error can't find the container with id b2187312e3363f2ee22790c5ed87330f51717dd17e3faf3918e1bb139dcbf2f2 Oct 30 13:12:55.654745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:55.654716 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t"] Oct 30 13:12:55.659315 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:55.659291 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3789b872_e323_4b81_a952_6783a7209ece.slice/crio-bf4b710cd3980ef55f3b23a9b8ea8de6df2aa0e57916756ba21c52010a959e1b WatchSource:0}: Error finding container bf4b710cd3980ef55f3b23a9b8ea8de6df2aa0e57916756ba21c52010a959e1b: Status 404 returned error can't find the container with id bf4b710cd3980ef55f3b23a9b8ea8de6df2aa0e57916756ba21c52010a959e1b Oct 30 13:12:56.225261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.225227 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" event={"ID":"be62a5bc-0e09-4d0e-a514-14b28e9b98df","Type":"ContainerStarted","Data":"0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706"} Oct 30 13:12:56.225761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.225432 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:12:56.225761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.225461 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" event={"ID":"be62a5bc-0e09-4d0e-a514-14b28e9b98df","Type":"ContainerStarted","Data":"b2187312e3363f2ee22790c5ed87330f51717dd17e3faf3918e1bb139dcbf2f2"} Oct 30 13:12:56.249624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.226871 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" event={"ID":"3789b872-e323-4b81-a952-6783a7209ece","Type":"ContainerStarted","Data":"d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e"} Oct 30 13:12:56.263400 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.263339 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645c06d9-c5e7-41eb-ab62-88a69dd722d0" path="/var/lib/kubelet/pods/645c06d9-c5e7-41eb-ab62-88a69dd722d0/volumes" Oct 30 13:12:56.263753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.263730 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" event={"ID":"3789b872-e323-4b81-a952-6783a7209ece","Type":"ContainerStarted","Data":"bf4b710cd3980ef55f3b23a9b8ea8de6df2aa0e57916756ba21c52010a959e1b"} Oct 30 13:12:56.263871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.263785 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:12:56.274240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.274193 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" podStartSLOduration=2.274179157 podStartE2EDuration="2.274179157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:56.272983246 +0000 UTC m=+552.139601419" watchObservedRunningTime="2026-10-30 13:12:56.274179157 +0000 UTC m=+552.140797331" Oct 30 13:12:56.310563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.310515 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" podStartSLOduration=2.310501933 podStartE2EDuration="2.310501933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:56.309357484 +0000 UTC m=+552.175975657" watchObservedRunningTime="2026-10-30 13:12:56.310501933 +0000 UTC m=+552.177120105" Oct 30 13:12:56.630679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.627286 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:56.793694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:56.793643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:57.038636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:57.038117 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:57.118402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:57.118365 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 13:12:58.195999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.195972 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 13:12:58.559618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.559094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:58.695063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.695036 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f"] Oct 30 13:12:58.698259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.698231 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:58.699040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.699013 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:58.716762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.716719 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-52af49\"" Oct 30 13:12:58.717149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.717125 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f"] Oct 30 13:12:58.772001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.771948 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-52af49-dockercfg-5cm8h\"" Oct 30 13:12:58.787112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.787082 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww"] Oct 30 13:12:58.853718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.844352 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fz2\" (UniqueName: \"kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2\") pod \"payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:58.853718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.844483 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:58.882118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.882092 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww"] Oct 30 13:12:58.882408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.882386 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:58.950630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.947207 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:58.950630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.947296 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:58.950630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.947471 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fz2\" (UniqueName: \"kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2\") pod \"payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:58.950630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.947542 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b5l\" (UniqueName: \"kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l\") pod \"payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:58.950630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.948510 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:58.975916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:58.975889 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fz2\" (UniqueName: \"kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2\") pod \"payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:59.015004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.014848 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:12:59.059645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.056409 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b5l\" (UniqueName: \"kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l\") pod \"payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:59.059645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.056478 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:59.059888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.059851 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:59.086465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.083866 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:59.092713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.092660 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b5l\" (UniqueName: \"kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l\") pod \"payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:59.204841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.204808 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:12:59.277563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.277493 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:59.376846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.376806 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f"] Oct 30 13:12:59.377373 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:59.377345 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdb9cc2_807a_437e_96c7_454f224e067e.slice/crio-0f23fdda353792ec043c4176409fdd375f3024b9d5e099b02122f9d44ff26ec3 WatchSource:0}: Error finding container 0f23fdda353792ec043c4176409fdd375f3024b9d5e099b02122f9d44ff26ec3: Status 404 returned error can't find the container with id 0f23fdda353792ec043c4176409fdd375f3024b9d5e099b02122f9d44ff26ec3 Oct 30 13:12:59.558806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.558779 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww"] Oct 30 13:12:59.560895 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:12:59.560868 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb571d1ac_9679_4f13_957e_f87bf892c875.slice/crio-12aab7e0b6dfdc270c560a27ccde1d3dba6ee1d0d6c847e0b8377e533cbce13b WatchSource:0}: Error finding container 12aab7e0b6dfdc270c560a27ccde1d3dba6ee1d0d6c847e0b8377e533cbce13b: Status 404 returned error can't find the container with id 12aab7e0b6dfdc270c560a27ccde1d3dba6ee1d0d6c847e0b8377e533cbce13b Oct 30 13:12:59.705762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.705729 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:59.861860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.861829 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:59.996432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:12:59.996396 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:00.155133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.155102 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 13:13:00.155313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.155170 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 13:13:00.318073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.317977 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" event={"ID":"b571d1ac-9679-4f13-957e-f87bf892c875","Type":"ContainerStarted","Data":"6a256bf9d3b8c8563ba30bbd635df26a75f21168c8ad4edf11d69c56e799a9e7"} Oct 30 13:13:00.318073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.318040 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" event={"ID":"b571d1ac-9679-4f13-957e-f87bf892c875","Type":"ContainerStarted","Data":"12aab7e0b6dfdc270c560a27ccde1d3dba6ee1d0d6c847e0b8377e533cbce13b"} Oct 30 13:13:00.318565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.318182 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:13:00.319871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.319825 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" event={"ID":"fbdb9cc2-807a-437e-96c7-454f224e067e","Type":"ContainerStarted","Data":"744ca13c4555f220af2005cca159f4e3a582a45a19392f48734e4c4fb6f07403"} Oct 30 13:13:00.320010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.319897 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" event={"ID":"fbdb9cc2-807a-437e-96c7-454f224e067e","Type":"ContainerStarted","Data":"0f23fdda353792ec043c4176409fdd375f3024b9d5e099b02122f9d44ff26ec3"} Oct 30 13:13:00.320010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.319930 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:13:00.358372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.358311 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" podStartSLOduration=2.358279355 podStartE2EDuration="2.358279355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:00.357046118 +0000 UTC m=+556.223664292" watchObservedRunningTime="2026-10-30 13:13:00.358279355 +0000 UTC m=+556.224897528" Oct 30 13:13:00.395277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:00.395216 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" podStartSLOduration=2.395202393 podStartE2EDuration="2.395202393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:00.394457426 +0000 UTC m=+556.261075600" watchObservedRunningTime="2026-10-30 13:13:00.395202393 +0000 UTC m=+556.261820565" Oct 30 13:13:01.257960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:01.257918 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:01.412519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:01.408818 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:01.546204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:01.546057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:01.682193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:01.682159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.177346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:02.177315 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.318338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:02.318299 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.463448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:02.463356 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.198035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:03.198003 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.345864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:03.345794 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:04.366066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:04.366035 5693 generic.go:356] "Generic (PLEG): container finished" podID="984445f4-1af1-460d-a436-7d55601d2866" containerID="854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481" exitCode=0 Oct 30 13:13:04.366569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:04.366117 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" event={"ID":"984445f4-1af1-460d-a436-7d55601d2866","Type":"ContainerDied","Data":"854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481"} Oct 30 13:13:04.366682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:04.366664 5693 scope.go:122] "RemoveContainer" containerID="854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481" Oct 30 13:13:04.368086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:04.368052 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:04.368184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:04.368128 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:04.368184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:04.368165 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:05.039663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.039624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:05.187236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.184377 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:05.374544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.374509 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" event={"ID":"984445f4-1af1-460d-a436-7d55601d2866","Type":"ContainerStarted","Data":"2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a"} Oct 30 13:13:05.577192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.577155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:05.851649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.847959 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:13:05.852959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.852925 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:13:05.853086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:05.852992 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:13:06.187680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:06.187574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.354212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:06.354165 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.506741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:06.506705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.848672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:06.848017 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:13:06.848672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:06.848085 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:13:06.967876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:06.967839 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:07.263462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:07.263438 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 13:13:07.852125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:07.852081 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:13:07.852527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:07.852177 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:13:08.298126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.298098 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 13:13:08.408906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.408861 5693 generic.go:356] "Generic (PLEG): container finished" podID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerID="2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92" exitCode=0 Oct 30 13:13:08.409076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.408977 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" event={"ID":"b8b856a0-3adb-4403-a497-91b12ad5d85d","Type":"ContainerDied","Data":"2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92"} Oct 30 13:13:08.409317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.409303 5693 scope.go:122] "RemoveContainer" containerID="2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92" Oct 30 13:13:08.410464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.410436 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:08.410529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.410504 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:08.410564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.410536 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:08.758013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.757971 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:08.844454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.843986 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:13:08.844454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:08.844049 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:13:09.417997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:09.417957 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" event={"ID":"b8b856a0-3adb-4403-a497-91b12ad5d85d","Type":"ContainerStarted","Data":"9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca"} Oct 30 13:13:09.446619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:09.446123 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:13:09.446619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:09.446200 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:13:09.446619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:09.446512 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:13:09.847037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:09.846999 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:13:09.847322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:09.847290 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:13:10.070635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:10.069708 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:13:10.445023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:10.444987 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:13:10.445645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:10.445611 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:13:10.614259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:10.614191 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:10.845041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:10.845002 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:13:10.845328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:10.845298 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:13:11.373849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:11.373812 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 13:13:11.377401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:11.377350 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 13:13:11.455217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:11.455179 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:13:11.455893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:11.455844 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:13:11.864623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:11.857932 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:13:11.876627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:11.869827 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:13:12.189576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:12.189483 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:12.450684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:12.444147 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:13:12.450684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:12.444234 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:13:12.701823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:12.700721 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:13.446034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:13.445995 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:13:13.446316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:13.446287 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:13:14.093583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:14.093551 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:13:14.451086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:14.450989 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:13:14.451358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:14.451329 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:13:14.648165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:14.648130 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:14.841060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:14.841027 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.457521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:15.457486 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:13:15.459102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:15.459080 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 13:13:16.604389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:16.604345 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:17.230636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:17.230581 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:18.326004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:18.325969 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:18.470928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:18.470894 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.028816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.028783 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.200073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.200033 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.537274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.537239 5693 generic.go:356] "Generic (PLEG): container finished" podID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerID="614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4" exitCode=0 Oct 30 13:13:19.537274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.537267 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" event={"ID":"6b54d4a7-0815-4fa8-958e-67463f66156e","Type":"ContainerDied","Data":"614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4"} Oct 30 13:13:19.537788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.537778 5693 scope.go:122] "RemoveContainer" containerID="614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4" Oct 30 13:13:19.539280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.539248 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:19.539371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.539329 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:19.539371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:19.539366 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:20.136807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:20.130530 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:20.557495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:20.557413 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" event={"ID":"6b54d4a7-0815-4fa8-958e-67463f66156e","Type":"ContainerStarted","Data":"e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4"} Oct 30 13:13:20.917684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:20.917565 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.522143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:21.522105 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:13:21.523621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:21.523581 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:13:21.523748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:21.523644 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:13:21.586246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:21.586210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.739245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:21.739196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.975165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:21.975131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:22.524565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:22.524527 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:13:22.524790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:22.524634 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:13:23.078560 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:13:23.078523 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:13:23.521878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:23.521842 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:13:23.522078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:23.521905 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:13:23.745418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:23.745373 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:24.199787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:24.199753 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:13:24.521835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:24.521800 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:13:24.522031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:24.521858 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:13:25.436623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:25.433003 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.522696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:25.522665 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:13:25.522872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:25.522720 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:13:26.526567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:26.526534 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:13:26.527573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:26.527555 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:13:26.900343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:26.900263 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:27.036332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:27.036299 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:28.722321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:28.722280 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:30.409382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:30.409349 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:32.115634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:32.115583 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:33.710082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:33.710044 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:34.273139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:34.273069 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:35.327937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:35.327899 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:36.961956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:36.961920 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:38.631042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:38.630782 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:40.015568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:40.015532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:40.216866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:40.216824 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:41.916513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:41.916468 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:43.511920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:43.511880 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:45.158347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.158317 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98"] Oct 30 13:13:45.163991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.163967 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.180435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.180394 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98"] Oct 30 13:13:45.189789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.189756 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"openshift-service-ca.crt\"" Oct 30 13:13:45.189789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.189783 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"default-dockercfg-hlqhx\"" Oct 30 13:13:45.192171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.192139 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"kube-root-ca.crt\"" Oct 30 13:13:45.192171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.192155 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"premium-w2-ec41e5-kserve-self-signed-certs\"" Oct 30 13:13:45.218892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.218859 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.219064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.218913 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.219064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.219001 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.219167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.219110 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkfh\" (UniqueName: \"kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.219167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.219161 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.219290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.219270 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.226553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.226529 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:45.320707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.320672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.320906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.320748 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.320906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.320842 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.320906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.320895 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.321096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.320929 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.321096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.320998 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkfh\" (UniqueName: \"kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.321329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.321305 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.324488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.324430 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.328065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.327990 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.328700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.328661 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.343470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.343396 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.362635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.357203 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkfh\" (UniqueName: \"kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh\") pod \"premium-w2-ec41e5-kserve-6c5f6755dc-qrl98\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.362635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.361957 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:45.483699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.483672 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:45.801366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:45.801283 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98"] Oct 30 13:13:45.804395 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:13:45.804369 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1990209a_3cb1_4793_9d68_ae869455f487.slice/crio-40bc1a8efc67a5a35b42dbe39181c29832e459594fbdf655a3995b70e5bbdf48 WatchSource:0}: Error finding container 40bc1a8efc67a5a35b42dbe39181c29832e459594fbdf655a3995b70e5bbdf48: Status 404 returned error can't find the container with id 40bc1a8efc67a5a35b42dbe39181c29832e459594fbdf655a3995b70e5bbdf48 Oct 30 13:13:46.702977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:46.702939 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" event={"ID":"1990209a-3cb1-4793-9d68-ae869455f487","Type":"ContainerStarted","Data":"ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9"} Oct 30 13:13:46.703371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:46.702986 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" event={"ID":"1990209a-3cb1-4793-9d68-ae869455f487","Type":"ContainerStarted","Data":"40bc1a8efc67a5a35b42dbe39181c29832e459594fbdf655a3995b70e5bbdf48"} Oct 30 13:13:46.742900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:46.742846 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podStartSLOduration=1.742827192 podStartE2EDuration="1.742827192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:46.740315023 +0000 UTC m=+602.606933195" watchObservedRunningTime="2026-10-30 13:13:46.742827192 +0000 UTC m=+602.609445366" Oct 30 13:13:46.923773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:46.923741 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:47.484506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:47.484468 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:48.509367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:48.509328 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:50.208025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:50.207993 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:50.754109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:50.754060 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:51.841090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:51.841050 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:51.980274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:51.980246 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:53.055027 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:13:53.054986 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:13:53.412582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:53.412506 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:53.571083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:53.571043 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:55.128534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:55.128484 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:55.484125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:55.484091 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:55.496791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:55.496766 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:55.762778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:55.762695 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:13:56.835086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:56.835047 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:58.640587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:13:58.640526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:00.328629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:00.328506 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:01.955704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:01.955672 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:03.367298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:03.367267 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:03.742471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:03.742428 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:05.414266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:05.414226 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:07.101226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:07.101193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:08.811197 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:08.811166 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:10.427868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:10.427825 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:11.301109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.301079 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p"] Oct 30 13:14:11.306884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.306854 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.323743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.323716 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p"] Oct 30 13:14:11.331676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.331647 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"kube-root-ca.crt\"" Oct 30 13:14:11.331676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.331647 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"openshift-service-ca.crt\"" Oct 30 13:14:11.331889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.331767 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"premium-w1-52af49-kserve-self-signed-certs\"" Oct 30 13:14:11.334674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.334653 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"default-dockercfg-x88rn\"" Oct 30 13:14:11.361475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.361447 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.361638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.361564 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.361728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.361643 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.361804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.361762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.361938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.361913 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9t58\" (UniqueName: \"kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.362012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.361952 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.408729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.408699 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:11.463264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463226 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9t58\" (UniqueName: \"kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463416 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463463 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463516 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463546 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463736 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.463773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463773 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.464031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.463860 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.465704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.465686 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.466048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.466031 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.482666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.482638 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9t58\" (UniqueName: \"kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58\") pod \"premium-w1-52af49-kserve-5b9df45d4d-lrp4p\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.625317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.625234 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:11.923922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:11.923893 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p"] Oct 30 13:14:11.926876 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:14:11.926845 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode843b93f_d34c_482f_a433_55576dd790c1.slice/crio-888f2c3161b2dada640f949b9687114706c3b4844cfec205b5a6cd073cbfc83e WatchSource:0}: Error finding container 888f2c3161b2dada640f949b9687114706c3b4844cfec205b5a6cd073cbfc83e: Status 404 returned error can't find the container with id 888f2c3161b2dada640f949b9687114706c3b4844cfec205b5a6cd073cbfc83e Oct 30 13:14:12.096617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:12.096569 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:12.850308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:12.850275 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" event={"ID":"e843b93f-d34c-482f-a433-55576dd790c1","Type":"ContainerStarted","Data":"61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25"} Oct 30 13:14:12.850732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:12.850326 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" event={"ID":"e843b93f-d34c-482f-a433-55576dd790c1","Type":"ContainerStarted","Data":"888f2c3161b2dada640f949b9687114706c3b4844cfec205b5a6cd073cbfc83e"} Oct 30 13:14:12.889506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:12.889456 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podStartSLOduration=1.8894388229999999 podStartE2EDuration="1.889438823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:12.887072859 +0000 UTC m=+628.753691031" watchObservedRunningTime="2026-10-30 13:14:12.889438823 +0000 UTC m=+628.756056999" Oct 30 13:14:13.626011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:13.625980 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:13.795139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:13.795102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:15.419997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:15.419959 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:17.137285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:17.137248 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:18.830489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:18.830447 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:20.512257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:20.512218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:21.625685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:21.625644 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:21.638809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:21.638782 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:21.921306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:21.921227 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:14:22.270619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:22.270570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:23.055907 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:14:23.055867 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:14:23.921990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:23.921949 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:25.639469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:25.635271 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:27.328844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:27.328810 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:29.054232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:29.054192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:30.734714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:30.734550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:32.631289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:32.627935 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:34.451923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:34.451878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:35.446826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.446789 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc"] Oct 30 13:14:35.450819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.450792 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.479842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.479807 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5852ff-openshift-default-dockercfg-76pkk\"" Oct 30 13:14:35.480283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480233 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480282 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480336 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6gn\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480378 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480429 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480459 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480487 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.480765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.480507 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.484499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.484469 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc"] Oct 30 13:14:35.581479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581439 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6gn\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.581665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581496 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.581665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581523 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.581665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581545 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.581665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.581665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581589 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.581665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581644 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581732 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.581775 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.582000 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.582073 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.582242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.582322 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.582432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.582411 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.583970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.583948 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.584206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.584188 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.600724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.600699 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.600855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.600834 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6gn\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:35.765262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:35.765222 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:36.075647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.074432 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:36.082204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.082173 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc"] Oct 30 13:14:36.086310 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:14:36.086282 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a0a847_3d24_4406_b285_026bf3db0c77.slice/crio-a352a291d0464c3f2da8044817ed12325312381ca9ad28be68ace5fcd81800fc WatchSource:0}: Error finding container a352a291d0464c3f2da8044817ed12325312381ca9ad28be68ace5fcd81800fc: Status 404 returned error can't find the container with id a352a291d0464c3f2da8044817ed12325312381ca9ad28be68ace5fcd81800fc Oct 30 13:14:36.088731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.088695 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:14:36.088816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.088766 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:14:36.088816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.088797 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:14:36.994765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.994727 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" event={"ID":"d8a0a847-3d24-4406-b285-026bf3db0c77","Type":"ContainerStarted","Data":"9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56"} Oct 30 13:14:36.995179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:36.994785 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" event={"ID":"d8a0a847-3d24-4406-b285-026bf3db0c77","Type":"ContainerStarted","Data":"a352a291d0464c3f2da8044817ed12325312381ca9ad28be68ace5fcd81800fc"} Oct 30 13:14:37.765567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:37.765536 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:37.767372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:37.767345 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:14:37.767517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:37.767401 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:14:37.850623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:37.850150 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:38.766214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:38.766178 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:14:38.766637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:38.766229 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:14:39.655621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:39.650266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:39.766166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:39.766131 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:14:39.766337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:39.766195 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:14:40.772056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:40.772014 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:14:40.772546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:40.772097 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:14:41.770943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:41.769461 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:14:41.770943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:41.769537 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:14:41.803739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:41.803701 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:43.299029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:43.298339 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:43.299526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:43.299075 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:43.300863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:43.300690 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:14:43.356885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:43.356836 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podStartSLOduration=8.356823183 podStartE2EDuration="8.356823183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:37.052985321 +0000 UTC m=+652.919603497" watchObservedRunningTime="2026-10-30 13:14:43.356823183 +0000 UTC m=+659.223441427" Oct 30 13:14:43.652613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:43.652509 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:45.528500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:45.528457 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:47.153961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:47.153910 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:48.830679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:48.830643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:50.540173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:50.540131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:52.019316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:52.019283 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:52.156951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:52.156919 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:53.056617 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:14:53.056548 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:14:53.925335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:53.925246 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:55.611074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:55.611034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:57.320454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:57.320280 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:59.041717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:14:59.041676 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:00.196027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.195988 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5"] Oct 30 13:15:00.200486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.200457 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.201104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.201079 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g"] Oct 30 13:15:00.205320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.205300 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.222428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.222402 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\"" Oct 30 13:15:00.238163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.238133 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5"] Oct 30 13:15:00.244399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.244368 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g"] Oct 30 13:15:00.244517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.244443 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8"] Oct 30 13:15:00.249285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.249260 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.259714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.259690 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8"] Oct 30 13:15:00.335768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.335726 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.335928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.335804 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lccms\" (UniqueName: \"kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms\") pod \"maas-api-key-cleanup-29889435-9gch8\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.336123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.336099 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.336280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.336264 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-9gch8\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.336360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.336287 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzbvk\" (UniqueName: \"kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.336360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.336314 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzszl\" (UniqueName: \"kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.437793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.437756 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-9gch8\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.437958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.437805 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzbvk\" (UniqueName: \"kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.437958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.437847 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzszl\" (UniqueName: \"kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.438100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.437955 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.438100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.438002 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lccms\" (UniqueName: \"kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms\") pod \"maas-api-key-cleanup-29889435-9gch8\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.438100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.438059 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.438563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.438538 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-9gch8\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.438767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.438746 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.438872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.438849 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.458921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.458860 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzbvk\" (UniqueName: \"kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.464856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.464828 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lccms\" (UniqueName: \"kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms\") pod \"maas-api-key-cleanup-29889435-9gch8\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.465814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.465796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzszl\" (UniqueName: \"kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.519266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.519235 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:00.529074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.529044 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:00.566304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.566273 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:00.746624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.746503 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:00.821970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.821935 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5"] Oct 30 13:15:00.824563 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:15:00.824533 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35276766_9ef9_4b52_8fce_15461cc4a1ec.slice/crio-2b525f917467f5100b71bb9ff4256e81dde94d3538cb12f245444bc5e1c12b8b WatchSource:0}: Error finding container 2b525f917467f5100b71bb9ff4256e81dde94d3538cb12f245444bc5e1c12b8b: Status 404 returned error can't find the container with id 2b525f917467f5100b71bb9ff4256e81dde94d3538cb12f245444bc5e1c12b8b Oct 30 13:15:00.835335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.835288 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g"] Oct 30 13:15:00.837726 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:15:00.837701 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07f0fb0_9954_413d_980a_60ed93566041.slice/crio-3da5a81f4fd8c5d95fac81c8baedff1068896e80356f6e56cac858d863125ad5 WatchSource:0}: Error finding container 3da5a81f4fd8c5d95fac81c8baedff1068896e80356f6e56cac858d863125ad5: Status 404 returned error can't find the container with id 3da5a81f4fd8c5d95fac81c8baedff1068896e80356f6e56cac858d863125ad5 Oct 30 13:15:00.901169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:00.901135 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8"] Oct 30 13:15:00.903871 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:15:00.903834 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579869a7_407a_4e03_802f_0ac4b5fb8366.slice/crio-685734fb9ddbe36d023be5e09b4f9d6aa7238b98af182d93ecdb05aa42a686e9 WatchSource:0}: Error finding container 685734fb9ddbe36d023be5e09b4f9d6aa7238b98af182d93ecdb05aa42a686e9: Status 404 returned error can't find the container with id 685734fb9ddbe36d023be5e09b4f9d6aa7238b98af182d93ecdb05aa42a686e9 Oct 30 13:15:01.140034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:01.139943 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" event={"ID":"e07f0fb0-9954-413d-980a-60ed93566041","Type":"ContainerStarted","Data":"3da5a81f4fd8c5d95fac81c8baedff1068896e80356f6e56cac858d863125ad5"} Oct 30 13:15:01.141183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:01.141155 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerStarted","Data":"2b525f917467f5100b71bb9ff4256e81dde94d3538cb12f245444bc5e1c12b8b"} Oct 30 13:15:01.142336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:01.142311 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" event={"ID":"579869a7-407a-4e03-802f-0ac4b5fb8366","Type":"ContainerStarted","Data":"685734fb9ddbe36d023be5e09b4f9d6aa7238b98af182d93ecdb05aa42a686e9"} Oct 30 13:15:02.549628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:02.548257 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:04.165449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:04.165372 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" event={"ID":"e07f0fb0-9954-413d-980a-60ed93566041","Type":"ContainerStarted","Data":"1d1e2c1b1ccf18e7be2cbe5e0f7a3b8eaf9b2d0aade6487a2a0fd331e31bd919"} Oct 30 13:15:04.167179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:04.167152 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerStarted","Data":"c92cb1f4834ef074aca34038f74552c3b95b8e7bb77c57640ee85ea2be5d09ab"} Oct 30 13:15:04.205709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:04.205656 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" podStartSLOduration=1.391329325 podStartE2EDuration="4.205642912s" totalImagesPullingTime="2.814313587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.204499504 +0000 UTC m=+680.071117677" watchObservedRunningTime="2026-10-30 13:15:04.205642912 +0000 UTC m=+680.072261063" Oct 30 13:15:04.247220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:04.247182 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:04.266699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:04.266637 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" podStartSLOduration=1.213942958 podStartE2EDuration="4.2665811s" totalImagesPullingTime="3.052638142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.265784968 +0000 UTC m=+680.132403142" watchObservedRunningTime="2026-10-30 13:15:04.2665811 +0000 UTC m=+680.133199273" Oct 30 13:15:05.175215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.175174 5693 generic.go:356] "Generic (PLEG): container finished" podID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerID="45415fc6dbd9d57173e1cce12dc4cb5269a6fdaf5730a2e8b1fb948504f38e64" exitCode=60 Oct 30 13:15:05.175713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.175263 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" event={"ID":"579869a7-407a-4e03-802f-0ac4b5fb8366","Type":"ContainerDied","Data":"45415fc6dbd9d57173e1cce12dc4cb5269a6fdaf5730a2e8b1fb948504f38e64"} Oct 30 13:15:05.175713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.175615 5693 scope.go:122] "RemoveContainer" containerID="45415fc6dbd9d57173e1cce12dc4cb5269a6fdaf5730a2e8b1fb948504f38e64" Oct 30 13:15:05.177168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.177139 5693 generic.go:356] "Generic (PLEG): container finished" podID="e07f0fb0-9954-413d-980a-60ed93566041" containerID="1d1e2c1b1ccf18e7be2cbe5e0f7a3b8eaf9b2d0aade6487a2a0fd331e31bd919" exitCode=60 Oct 30 13:15:05.177308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.177273 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" event={"ID":"e07f0fb0-9954-413d-980a-60ed93566041","Type":"ContainerDied","Data":"1d1e2c1b1ccf18e7be2cbe5e0f7a3b8eaf9b2d0aade6487a2a0fd331e31bd919"} Oct 30 13:15:05.177489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.177473 5693 scope.go:122] "RemoveContainer" containerID="1d1e2c1b1ccf18e7be2cbe5e0f7a3b8eaf9b2d0aade6487a2a0fd331e31bd919" Oct 30 13:15:05.179055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.179035 5693 generic.go:356] "Generic (PLEG): container finished" podID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerID="c92cb1f4834ef074aca34038f74552c3b95b8e7bb77c57640ee85ea2be5d09ab" exitCode=60 Oct 30 13:15:05.179140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.179086 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerDied","Data":"c92cb1f4834ef074aca34038f74552c3b95b8e7bb77c57640ee85ea2be5d09ab"} Oct 30 13:15:05.179300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.179263 5693 scope.go:122] "RemoveContainer" containerID="c92cb1f4834ef074aca34038f74552c3b95b8e7bb77c57640ee85ea2be5d09ab" Oct 30 13:15:05.934570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:05.934528 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:06.186655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.186556 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerStarted","Data":"2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec"} Oct 30 13:15:06.188722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.188688 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" event={"ID":"579869a7-407a-4e03-802f-0ac4b5fb8366","Type":"ContainerStarted","Data":"f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f"} Oct 30 13:15:06.191275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.191243 5693 generic.go:356] "Generic (PLEG): container finished" podID="e07f0fb0-9954-413d-980a-60ed93566041" containerID="a845940ffb9770f2fb5dee46ea32525e9a91a722cb0a4ff2921d6f782ff30000" exitCode=60 Oct 30 13:15:06.191430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.191306 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" event={"ID":"e07f0fb0-9954-413d-980a-60ed93566041","Type":"ContainerDied","Data":"a845940ffb9770f2fb5dee46ea32525e9a91a722cb0a4ff2921d6f782ff30000"} Oct 30 13:15:06.191430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.191359 5693 scope.go:122] "RemoveContainer" containerID="1d1e2c1b1ccf18e7be2cbe5e0f7a3b8eaf9b2d0aade6487a2a0fd331e31bd919" Oct 30 13:15:06.191833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.191686 5693 scope.go:122] "RemoveContainer" containerID="a845940ffb9770f2fb5dee46ea32525e9a91a722cb0a4ff2921d6f782ff30000" Oct 30 13:15:06.192029 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:06.191990 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g_odh-ai-gateway-infra(e07f0fb0-9954-413d-980a-60ed93566041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" podUID="e07f0fb0-9954-413d-980a-60ed93566041" Oct 30 13:15:06.324579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:06.324527 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" podStartSLOduration=3.359968278 podStartE2EDuration="6.324514599s" totalImagesPullingTime="2.964546321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.323350259 +0000 UTC m=+682.189968431" watchObservedRunningTime="2026-10-30 13:15:06.324514599 +0000 UTC m=+682.191132772" Oct 30 13:15:07.198399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.198367 5693 scope.go:122] "RemoveContainer" containerID="a845940ffb9770f2fb5dee46ea32525e9a91a722cb0a4ff2921d6f782ff30000" Oct 30 13:15:07.198882 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:07.198729 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g_odh-ai-gateway-infra(e07f0fb0-9954-413d-980a-60ed93566041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" podUID="e07f0fb0-9954-413d-980a-60ed93566041" Oct 30 13:15:07.200044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.200011 5693 generic.go:356] "Generic (PLEG): container finished" podID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerID="2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec" exitCode=60 Oct 30 13:15:07.200178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.200093 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerDied","Data":"2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec"} Oct 30 13:15:07.200178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.200144 5693 scope.go:122] "RemoveContainer" containerID="c92cb1f4834ef074aca34038f74552c3b95b8e7bb77c57640ee85ea2be5d09ab" Oct 30 13:15:07.200340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.200321 5693 scope.go:122] "RemoveContainer" containerID="2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec" Oct 30 13:15:07.200631 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:07.200592 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5_odh-ai-gateway-infra(35276766-9ef9-4b52-8fce-15461cc4a1ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" Oct 30 13:15:07.202198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.202174 5693 generic.go:356] "Generic (PLEG): container finished" podID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerID="f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f" exitCode=60 Oct 30 13:15:07.202282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.202247 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" event={"ID":"579869a7-407a-4e03-802f-0ac4b5fb8366","Type":"ContainerDied","Data":"f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f"} Oct 30 13:15:07.202454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.202441 5693 scope.go:122] "RemoveContainer" containerID="f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f" Oct 30 13:15:07.202690 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:07.202673 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-9gch8_odh-ai-gateway-infra(579869a7-407a-4e03-802f-0ac4b5fb8366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" Oct 30 13:15:07.214833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.214803 5693 scope.go:122] "RemoveContainer" containerID="45415fc6dbd9d57173e1cce12dc4cb5269a6fdaf5730a2e8b1fb948504f38e64" Oct 30 13:15:07.691067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:07.691034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:08.211374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:08.211342 5693 scope.go:122] "RemoveContainer" containerID="2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec" Oct 30 13:15:08.211957 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:08.211731 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5_odh-ai-gateway-infra(35276766-9ef9-4b52-8fce-15461cc4a1ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" Oct 30 13:15:08.213411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:08.213390 5693 scope.go:122] "RemoveContainer" containerID="f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f" Oct 30 13:15:08.213666 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:08.213645 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-9gch8_odh-ai-gateway-infra(579869a7-407a-4e03-802f-0ac4b5fb8366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" Oct 30 13:15:09.465291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:09.465242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:09.939520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:09.939483 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:10.105205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:10.105163 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:10.763174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:10.763141 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:10.989466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:10.989432 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk"] Oct 30 13:15:10.994894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:10.994869 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.011327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.011295 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 13:15:11.011327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.011299 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 13:15:11.017113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.017048 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.017247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.017158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.017345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.017284 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.017345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.017339 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t98s\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.023355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.023326 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 13:15:11.023693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.023668 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1856bd8d477d\"" Oct 30 13:15:11.023693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.023682 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-132ab67347e8\"" Oct 30 13:15:11.023844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.023762 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 13:15:11.023844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.023814 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-b6xtq\"" Oct 30 13:15:11.026168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.026150 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk"] Oct 30 13:15:11.125302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.125260 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.125499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.125378 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.125499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.125487 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.125630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.125544 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t98s\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.127615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.127572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.128453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.128428 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.137973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.137895 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.149275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.149246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t98s\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s\") pod \"payload-processing-external-model-55d4cbbbf5-sckpk\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.244654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.244402 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:11.309222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.309125 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:11.633529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:11.633071 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk"] Oct 30 13:15:11.635981 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:15:11.635948 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc74e7b3_a349_49c9_bd81_533bc1c6e2cc.slice/crio-80846e7a9bf8c3bf379eb7a1cd44a6020cc1c13933b1ffe443b04400c0f38491 WatchSource:0}: Error finding container 80846e7a9bf8c3bf379eb7a1cd44a6020cc1c13933b1ffe443b04400c0f38491: Status 404 returned error can't find the container with id 80846e7a9bf8c3bf379eb7a1cd44a6020cc1c13933b1ffe443b04400c0f38491 Oct 30 13:15:12.311265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:12.311227 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" event={"ID":"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc","Type":"ContainerStarted","Data":"847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4"} Oct 30 13:15:12.311830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:12.311279 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" event={"ID":"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc","Type":"ContainerStarted","Data":"80846e7a9bf8c3bf379eb7a1cd44a6020cc1c13933b1ffe443b04400c0f38491"} Oct 30 13:15:12.311830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:12.311704 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:12.383354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:12.383284 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" podStartSLOduration=2.383261811 podStartE2EDuration="2.383261811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:12.360131486 +0000 UTC m=+688.226749672" watchObservedRunningTime="2026-10-30 13:15:12.383261811 +0000 UTC m=+688.249879987" Oct 30 13:15:13.043619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:13.042520 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:14.883520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:14.883482 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:16.711051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:16.710996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:18.521632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:18.517019 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:19.247238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:19.247178 5693 scope.go:122] "RemoveContainer" containerID="2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec" Oct 30 13:15:19.247432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:19.247294 5693 scope.go:122] "RemoveContainer" containerID="f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f" Oct 30 13:15:19.247527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:19.247508 5693 scope.go:122] "RemoveContainer" containerID="a845940ffb9770f2fb5dee46ea32525e9a91a722cb0a4ff2921d6f782ff30000" Oct 30 13:15:19.867379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:19.867287 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:20.023381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.023342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:20.177024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.176954 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:20.415043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.414953 5693 generic.go:356] "Generic (PLEG): container finished" podID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerID="089a044a125132afd919b75be863338980782f2a28c943d2366952936f132521" exitCode=60 Oct 30 13:15:20.415043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.415026 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerDied","Data":"089a044a125132afd919b75be863338980782f2a28c943d2366952936f132521"} Oct 30 13:15:20.415277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.415081 5693 scope.go:122] "RemoveContainer" containerID="2476381b2370840c6a2c58e2771c9224c58c5662841eaafd78a6b5ef13029aec" Oct 30 13:15:20.415465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.415443 5693 scope.go:122] "RemoveContainer" containerID="089a044a125132afd919b75be863338980782f2a28c943d2366952936f132521" Oct 30 13:15:20.415804 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:20.415779 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5_odh-ai-gateway-infra(35276766-9ef9-4b52-8fce-15461cc4a1ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" Oct 30 13:15:20.417147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.417129 5693 generic.go:356] "Generic (PLEG): container finished" podID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerID="985164e1fbc2a6db97e940582e931de65ab54a25960ea42228214e21e0d74c73" exitCode=60 Oct 30 13:15:20.417235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.417199 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" event={"ID":"579869a7-407a-4e03-802f-0ac4b5fb8366","Type":"ContainerDied","Data":"985164e1fbc2a6db97e940582e931de65ab54a25960ea42228214e21e0d74c73"} Oct 30 13:15:20.417545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.417529 5693 scope.go:122] "RemoveContainer" containerID="985164e1fbc2a6db97e940582e931de65ab54a25960ea42228214e21e0d74c73" Oct 30 13:15:20.417823 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:20.417800 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-9gch8_odh-ai-gateway-infra(579869a7-407a-4e03-802f-0ac4b5fb8366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" Oct 30 13:15:20.419096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.419076 5693 generic.go:356] "Generic (PLEG): container finished" podID="e07f0fb0-9954-413d-980a-60ed93566041" containerID="37f87b6317a1c379848f4bcfea9fe6d531fa8e3a23a4288148be8efb700e2227" exitCode=60 Oct 30 13:15:20.419192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.419127 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" event={"ID":"e07f0fb0-9954-413d-980a-60ed93566041","Type":"ContainerDied","Data":"37f87b6317a1c379848f4bcfea9fe6d531fa8e3a23a4288148be8efb700e2227"} Oct 30 13:15:20.419441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.419426 5693 scope.go:122] "RemoveContainer" containerID="37f87b6317a1c379848f4bcfea9fe6d531fa8e3a23a4288148be8efb700e2227" Oct 30 13:15:20.419661 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:20.419644 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g_odh-ai-gateway-infra(e07f0fb0-9954-413d-980a-60ed93566041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" podUID="e07f0fb0-9954-413d-980a-60ed93566041" Oct 30 13:15:20.426958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.426940 5693 scope.go:122] "RemoveContainer" containerID="f00849721a0f6fc10f875d245f5549d60cf103e4eb0e4b8c04a757b48eadfc1f" Oct 30 13:15:20.438445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:20.438429 5693 scope.go:122] "RemoveContainer" containerID="a845940ffb9770f2fb5dee46ea32525e9a91a722cb0a4ff2921d6f782ff30000" Oct 30 13:15:21.467456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.467426 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5"] Oct 30 13:15:21.502182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.502152 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8"] Oct 30 13:15:21.553471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.553439 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g"] Oct 30 13:15:21.629650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.629621 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:21.703038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.703013 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:21.737313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.737292 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:21.751699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.751671 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca\") pod \"35276766-9ef9-4b52-8fce-15461cc4a1ec\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " Oct 30 13:15:21.751850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.751715 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms\" (UniqueName: \"kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms\") pod \"579869a7-407a-4e03-802f-0ac4b5fb8366\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " Oct 30 13:15:21.751850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.751796 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk\" (UniqueName: \"kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk\") pod \"35276766-9ef9-4b52-8fce-15461cc4a1ec\" (UID: \"35276766-9ef9-4b52-8fce-15461cc4a1ec\") " Oct 30 13:15:21.752006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.751986 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca\") pod \"579869a7-407a-4e03-802f-0ac4b5fb8366\" (UID: \"579869a7-407a-4e03-802f-0ac4b5fb8366\") " Oct 30 13:15:21.752062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.752022 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca" pod "35276766-9ef9-4b52-8fce-15461cc4a1ec" (UID: "35276766-9ef9-4b52-8fce-15461cc4a1ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.752351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.752326 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca" pod "579869a7-407a-4e03-802f-0ac4b5fb8366" (UID: "579869a7-407a-4e03-802f-0ac4b5fb8366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.753965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.752642 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579869a7-407a-4e03-802f-0ac4b5fb8366-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.753965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.752668 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35276766-9ef9-4b52-8fce-15461cc4a1ec-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.753965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.753944 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms" pod "579869a7-407a-4e03-802f-0ac4b5fb8366" (UID: "579869a7-407a-4e03-802f-0ac4b5fb8366"). InnerVolumeSpecName "kube-api-access-lccms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.754289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.754258 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk" pod "35276766-9ef9-4b52-8fce-15461cc4a1ec" (UID: "35276766-9ef9-4b52-8fce-15461cc4a1ec"). InnerVolumeSpecName "kube-api-access-tzbvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.840040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.832248 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:21.862830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.853989 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl\" (UniqueName: \"kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl\") pod \"e07f0fb0-9954-413d-980a-60ed93566041\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " Oct 30 13:15:21.862830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.854090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca\") pod \"e07f0fb0-9954-413d-980a-60ed93566041\" (UID: \"e07f0fb0-9954-413d-980a-60ed93566041\") " Oct 30 13:15:21.862830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.855052 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lccms\" (UniqueName: \"kubernetes.io/projected/579869a7-407a-4e03-802f-0ac4b5fb8366-kube-api-access-lccms\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.862830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.855076 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzbvk\" (UniqueName: \"kubernetes.io/projected/35276766-9ef9-4b52-8fce-15461cc4a1ec-kube-api-access-tzbvk\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.862830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.860774 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca" pod "e07f0fb0-9954-413d-980a-60ed93566041" (UID: "e07f0fb0-9954-413d-980a-60ed93566041"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.867230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.867160 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl" pod "e07f0fb0-9954-413d-980a-60ed93566041" (UID: "e07f0fb0-9954-413d-980a-60ed93566041"). InnerVolumeSpecName "kube-api-access-mzszl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.956262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.956218 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzszl\" (UniqueName: \"kubernetes.io/projected/e07f0fb0-9954-413d-980a-60ed93566041-kube-api-access-mzszl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.956262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:21.956256 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07f0fb0-9954-413d-980a-60ed93566041-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:22.436338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.436305 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" Oct 30 13:15:22.436338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.436319 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8" event={"ID":"579869a7-407a-4e03-802f-0ac4b5fb8366","Type":"ContainerDied","Data":"685734fb9ddbe36d023be5e09b4f9d6aa7238b98af182d93ecdb05aa42a686e9"} Oct 30 13:15:22.436561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.436407 5693 scope.go:122] "RemoveContainer" containerID="985164e1fbc2a6db97e940582e931de65ab54a25960ea42228214e21e0d74c73" Oct 30 13:15:22.437721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.437693 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" event={"ID":"e07f0fb0-9954-413d-980a-60ed93566041","Type":"ContainerDied","Data":"3da5a81f4fd8c5d95fac81c8baedff1068896e80356f6e56cac858d863125ad5"} Oct 30 13:15:22.437855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.437706 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g" Oct 30 13:15:22.439063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.439040 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" event={"ID":"35276766-9ef9-4b52-8fce-15461cc4a1ec","Type":"ContainerDied","Data":"2b525f917467f5100b71bb9ff4256e81dde94d3538cb12f245444bc5e1c12b8b"} Oct 30 13:15:22.439158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.439065 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5" Oct 30 13:15:22.446782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.446754 5693 scope.go:122] "RemoveContainer" containerID="37f87b6317a1c379848f4bcfea9fe6d531fa8e3a23a4288148be8efb700e2227" Oct 30 13:15:22.454802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.454787 5693 scope.go:122] "RemoveContainer" containerID="089a044a125132afd919b75be863338980782f2a28c943d2366952936f132521" Oct 30 13:15:22.486002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.485978 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8"] Oct 30 13:15:22.497622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.497586 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-9gch8"] Oct 30 13:15:22.529002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.528932 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5"] Oct 30 13:15:22.531474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.531445 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889435-g25z5"] Oct 30 13:15:22.564827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.564796 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g"] Oct 30 13:15:22.568401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:22.568378 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889435-4mz9g"] Oct 30 13:15:23.054689 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:23.054650 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:15:23.321468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:23.321393 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:23.507648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:23.507516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:24.254212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:24.254179 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" path="/var/lib/kubelet/pods/35276766-9ef9-4b52-8fce-15461cc4a1ec/volumes" Oct 30 13:15:24.254649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:24.254635 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" path="/var/lib/kubelet/pods/579869a7-407a-4e03-802f-0ac4b5fb8366/volumes" Oct 30 13:15:24.255000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:24.254987 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07f0fb0-9954-413d-980a-60ed93566041" path="/var/lib/kubelet/pods/e07f0fb0-9954-413d-980a-60ed93566041/volumes" Oct 30 13:15:25.105240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105194 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4"] Oct 30 13:15:25.105783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105764 5693 cpu_manager.go:414] "containerMap: removing container" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.105832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105787 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.105832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105806 5693 cpu_manager.go:414] "containerMap: removing container" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.105832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105813 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.105832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105820 5693 cpu_manager.go:414] "containerMap: removing container" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.105832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105825 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.105832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105833 5693 cpu_manager.go:414] "containerMap: removing container" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105838 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105852 5693 cpu_manager.go:414] "containerMap: removing container" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105857 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105879 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105887 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105895 5693 cpu_manager.go:414] "containerMap: removing container" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105899 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105907 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105912 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105918 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.105923 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106015 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106026 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106033 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106042 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07f0fb0-9954-413d-980a-60ed93566041" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106054 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106060 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106066 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:25.106291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.106073 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="35276766-9ef9-4b52-8fce-15461cc4a1ec" containerName="cleanup" Oct 30 13:15:25.111142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.111123 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.142329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.142293 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4"] Oct 30 13:15:25.187219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.187182 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:25.191462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191434 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191479 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdnm\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191511 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191557 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191584 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191758 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191918 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.191974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.191952 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.292760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.292728 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.292953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.292793 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.292953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.292823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdnm\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.292953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.292858 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.292953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.292906 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.292953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.292947 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293030 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293052 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293116 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293221 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293245 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293297 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.293710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.293689 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.295348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.295324 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.295681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.295661 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.312051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.312026 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.312243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.312211 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdnm\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm\") pod \"maas-default-gateway-openshift-default-67bf986f99-zklq4\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.424923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.424838 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:25.749416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.749383 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4"] Oct 30 13:15:25.752785 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:15:25.752749 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b1c585_2d9d_498a_b1ed_d80e5ad5e432.slice/crio-e387db2331b1b2e893e8767eaed3077cfd2dadbe5a4c5bba44de79b66de1ecd3 WatchSource:0}: Error finding container e387db2331b1b2e893e8767eaed3077cfd2dadbe5a4c5bba44de79b66de1ecd3: Status 404 returned error can't find the container with id e387db2331b1b2e893e8767eaed3077cfd2dadbe5a4c5bba44de79b66de1ecd3 Oct 30 13:15:25.755356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.755320 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:15:25.755458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.755399 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:15:25.755458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:25.755431 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:15:26.487645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:26.485912 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" event={"ID":"40b1c585-2d9d-498a-b1ed-d80e5ad5e432","Type":"ContainerStarted","Data":"702ae4e3e2878c2ba1ec7e4b64a2a5d347b1453b21fb85c162333e35aad1456c"} Oct 30 13:15:26.487645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:26.486020 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" event={"ID":"40b1c585-2d9d-498a-b1ed-d80e5ad5e432","Type":"ContainerStarted","Data":"e387db2331b1b2e893e8767eaed3077cfd2dadbe5a4c5bba44de79b66de1ecd3"} Oct 30 13:15:26.545712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:26.545505 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podStartSLOduration=1.545485132 podStartE2EDuration="1.545485132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:26.542104435 +0000 UTC m=+702.408722613" watchObservedRunningTime="2026-10-30 13:15:26.545485132 +0000 UTC m=+702.412103308" Oct 30 13:15:27.117685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:27.117644 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:27.431677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:27.430798 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:27.431677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:27.430943 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:27.431677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:27.430986 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:28.432088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:28.432046 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:28.432717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:28.432682 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:29.426070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:29.426029 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:29.426265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:29.426097 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:29.689009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:29.688932 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:29.897821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:29.897785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:30.425874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:30.425842 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:30.426050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:30.425897 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:30.753826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:30.753790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:31.426178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:31.426137 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:31.426334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:31.426199 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:31.717083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:31.716995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:32.426177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:32.426139 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:32.426583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:32.426224 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:33.404873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:33.404837 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:33.435191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:33.435154 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:15:33.435920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:33.435789 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 13:15:34.768061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:34.768027 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:34.768634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:34.768618 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:34.769524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:34.769501 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:15:35.073302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:35.073156 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:36.254056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:36.254023 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:36.820562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:36.820524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:39.068554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:39.068516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:40.902222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:40.902186 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:42.646178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:42.646131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:44.042169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:44.042132 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk"] Oct 30 13:15:44.303158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:44.303067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:45.414179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:45.414144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:46.101310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:46.101277 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:47.875396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:47.875358 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:49.043579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.043514 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" podUID="fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" containerName="payload-processing" containerID="cri-o://847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4" gracePeriod=25 Oct 30 13:15:49.292718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.292692 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:49.325384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.325328 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume\") pod \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " Oct 30 13:15:49.325494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.325393 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s\") pod \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " Oct 30 13:15:49.325494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.325428 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume\") pod \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " Oct 30 13:15:49.325571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.325563 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials\") pod \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\" (UID: \"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc\") " Oct 30 13:15:49.325825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.325794 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume" pod "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" (UID: "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:49.325946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.325901 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume" pod "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" (UID: "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:49.326072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.326034 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:49.326072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.326057 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:49.327841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.327818 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s" pod "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" (UID: "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc"). InnerVolumeSpecName "kube-api-access-8t98s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:49.327950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.327901 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials" pod "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" (UID: "fc74e7b3-a349-49c9-bd81-533bc1c6e2cc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:49.427296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.427264 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:49.427296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.427292 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t98s\" (UniqueName: \"kubernetes.io/projected/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc-kube-api-access-8t98s\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:15:49.598058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.597974 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:49.717690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.717650 5693 generic.go:356] "Generic (PLEG): container finished" podID="fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" containerID="847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4" exitCode=0 Oct 30 13:15:49.717864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.717700 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" Oct 30 13:15:49.717864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.717709 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" event={"ID":"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc","Type":"ContainerDied","Data":"847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4"} Oct 30 13:15:49.717864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.717759 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk" event={"ID":"fc74e7b3-a349-49c9-bd81-533bc1c6e2cc","Type":"ContainerDied","Data":"80846e7a9bf8c3bf379eb7a1cd44a6020cc1c13933b1ffe443b04400c0f38491"} Oct 30 13:15:49.717864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.717782 5693 scope.go:122] "RemoveContainer" containerID="847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4" Oct 30 13:15:49.728076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.728059 5693 scope.go:122] "RemoveContainer" containerID="847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4" Oct 30 13:15:49.728338 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:49.728315 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4\": container with ID starting with 847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4 not found: ID does not exist" containerID="847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4" Oct 30 13:15:49.728408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.728349 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4"} err="failed to get container status \"847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4\": rpc error: code = NotFound desc = could not find container \"847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4\": container with ID starting with 847e45b062ed5a131c7bfa54ea119b3fb630e2e037d2807b239f4e9827c2dcf4 not found: ID does not exist" Oct 30 13:15:49.768344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.768320 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk"] Oct 30 13:15:49.772418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:49.772396 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-sckpk"] Oct 30 13:15:50.253716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.253674 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" path="/var/lib/kubelet/pods/fc74e7b3-a349-49c9-bd81-533bc1c6e2cc/volumes" Oct 30 13:15:50.771066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.771039 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77"] Oct 30 13:15:50.771669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.771650 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" containerName="payload-processing" Oct 30 13:15:50.771669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.771670 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" containerName="payload-processing" Oct 30 13:15:50.771979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.771876 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="579869a7-407a-4e03-802f-0ac4b5fb8366" containerName="cleanup" Oct 30 13:15:50.771979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.771897 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc74e7b3-a349-49c9-bd81-533bc1c6e2cc" containerName="payload-processing" Oct 30 13:15:50.776692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.776673 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.805187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.805153 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77"] Oct 30 13:15:50.841675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.841642 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.841831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.841716 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.841831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.841752 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.841831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.841826 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zds7\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.841953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.841856 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.841953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.841883 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.842025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.842007 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.842061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.842039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.842171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.842149 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.951296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.951251 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.952747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.952719 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.952985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.952968 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.953084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.952025 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.953959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.953102 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.954176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.954157 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.966398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.953893 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.967276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.966579 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.979346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.968347 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.979346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.967247 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.979346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.975464 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.979346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.979296 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.980342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.980007 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.980342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.980308 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.980917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.980878 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.981127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.981024 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:50.989080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:50.982682 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zds7\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:51.004617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.002862 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zds7\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:51.090714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.090587 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:51.425000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.424964 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77"] Oct 30 13:15:51.427017 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:15:51.426978 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5163624d_38bf_425b_bb38_dd6f1b14668a.slice/crio-c0015a3abaf14c317a6d3413253e1ee2331245100e1b538af247a3686ca3354d WatchSource:0}: Error finding container c0015a3abaf14c317a6d3413253e1ee2331245100e1b538af247a3686ca3354d: Status 404 returned error can't find the container with id c0015a3abaf14c317a6d3413253e1ee2331245100e1b538af247a3686ca3354d Oct 30 13:15:51.431437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.431363 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:15:51.431580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.431479 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:15:51.431580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.431531 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:15:51.435850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.435823 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:51.738162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.737727 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" event={"ID":"5163624d-38bf-425b-bb38-dd6f1b14668a","Type":"ContainerStarted","Data":"9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10"} Oct 30 13:15:51.738162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:51.737816 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" event={"ID":"5163624d-38bf-425b-bb38-dd6f1b14668a","Type":"ContainerStarted","Data":"c0015a3abaf14c317a6d3413253e1ee2331245100e1b538af247a3686ca3354d"} Oct 30 13:15:52.093677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:52.092732 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:15:52.093841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:52.093808 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:52.093922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:52.093857 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:52.185621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:52.185099 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:53.099618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:53.099001 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:53.099618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:53.099082 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:53.182059 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:15:53.182013 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:15:54.102056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:54.102009 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:54.102678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:54.102640 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:54.139191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:54.139155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:55.091905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:55.091824 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:55.091905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:55.091879 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:55.753566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:55.753512 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:56.091993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:56.091895 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:56.091993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:56.091974 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:56.123418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:56.123309 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:57.091050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:57.091013 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:57.091661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:57.091076 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:57.918169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:57.918138 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:58.091640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:58.091584 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:58.092040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:58.091662 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:59.091900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:59.091863 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:15:59.092314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:59.091952 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 13:15:59.823473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:15:59.823431 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:00.095206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:00.095123 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:16:00.095704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:00.095422 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:16:00.096139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:00.096122 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:16:00.150972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:00.150923 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podStartSLOduration=10.150908726 podStartE2EDuration="10.150908726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:51.786800025 +0000 UTC m=+727.653418211" watchObservedRunningTime="2026-10-30 13:16:00.150908726 +0000 UTC m=+736.017526900" Oct 30 13:16:01.436621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:01.428358 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:03.045200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:03.045078 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:04.824088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:04.824040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:05.273932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.273897 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r"] Oct 30 13:16:05.277959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.277937 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.292439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.292417 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9bf8ad-openshift-default-dockercfg-shhh4\"" Oct 30 13:16:05.307054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.307029 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r"] Oct 30 13:16:05.351915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.351881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.351947 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352068 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352146 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352184 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352203 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47d6\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352261 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352324 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.352347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.352342 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453490 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453527 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453588 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453662 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453696 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453719 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453758 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.453817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453786 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47d6\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.454150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.453831 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.454217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.454195 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.454320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.454290 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.454388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.454345 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.454388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.454359 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.454483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.454401 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.455983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.455963 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.456206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.456188 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.471936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.471914 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.472073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.472056 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47d6\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.592906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.592827 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:05.907976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.907938 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r"] Oct 30 13:16:05.910967 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:16:05.910939 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960c70b8_eefa_4af4_af60_6b509db15b39.slice/crio-e352ebafea352dfd78fcdfe0ee01a63f007e18771f5452575db48198638028e1 WatchSource:0}: Error finding container e352ebafea352dfd78fcdfe0ee01a63f007e18771f5452575db48198638028e1: Status 404 returned error can't find the container with id e352ebafea352dfd78fcdfe0ee01a63f007e18771f5452575db48198638028e1 Oct 30 13:16:05.913512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.913479 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:05.913619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.913546 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:05.913619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.913579 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:05.919341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:05.919312 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" event={"ID":"960c70b8-eefa-4af4-af60-6b509db15b39","Type":"ContainerStarted","Data":"e352ebafea352dfd78fcdfe0ee01a63f007e18771f5452575db48198638028e1"} Oct 30 13:16:06.600725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:06.600682 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:06.949688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:06.926636 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" event={"ID":"960c70b8-eefa-4af4-af60-6b509db15b39","Type":"ContainerStarted","Data":"be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c"} Oct 30 13:16:07.023177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:07.023121 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podStartSLOduration=2.023106988 podStartE2EDuration="2.023106988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:07.011843676 +0000 UTC m=+742.878461848" watchObservedRunningTime="2026-10-30 13:16:07.023106988 +0000 UTC m=+742.889725160" Oct 30 13:16:07.593324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:07.593288 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:07.594996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:07.594970 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:16:07.595060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:07.595037 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:16:08.354623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:08.354191 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.594178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:08.594138 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:16:08.594374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:08.594209 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:16:08.789001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:08.788920 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.945621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:08.945033 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:09.154630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.154511 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9"] Oct 30 13:16:09.159077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.159053 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.173530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.173502 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9"] Oct 30 13:16:09.181717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.181689 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 13:16:09.181838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.181699 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 13:16:09.182944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.182923 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1856bd8d477d\"" Oct 30 13:16:09.183048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.182925 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 13:16:09.187726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.187703 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-b6xtq\"" Oct 30 13:16:09.187726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.187712 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 13:16:09.187884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.187769 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-132ab67347e8\"" Oct 30 13:16:09.208719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.208657 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.208719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.208702 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtj6\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.208987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.208868 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.208987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.208924 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.309938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.309899 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.310104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.309953 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.310198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.310168 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.310260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.310220 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtj6\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.310684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.310659 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.310792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.310704 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.312487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.312465 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.352409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.352371 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtj6\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6\") pod \"payload-processing-external-model-55d4cbbbf5-hg9m9\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.485624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.475148 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:09.593523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.593297 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:16:09.593523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.593354 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:16:09.860865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.857634 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9"] Oct 30 13:16:09.861823 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:16:09.861773 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ace2e92_51cc_4bee_b050_b8138b997975.slice/crio-6064088b565e2c298ae65a1ac61e3245fdc0bb622578cdd651f7b15a039a8ac0 WatchSource:0}: Error finding container 6064088b565e2c298ae65a1ac61e3245fdc0bb622578cdd651f7b15a039a8ac0: Status 404 returned error can't find the container with id 6064088b565e2c298ae65a1ac61e3245fdc0bb622578cdd651f7b15a039a8ac0 Oct 30 13:16:09.995118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.995077 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" event={"ID":"2ace2e92-51cc-4bee-b050-b8138b997975","Type":"ContainerStarted","Data":"90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3"} Oct 30 13:16:09.995294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.995143 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" event={"ID":"2ace2e92-51cc-4bee-b050-b8138b997975","Type":"ContainerStarted","Data":"6064088b565e2c298ae65a1ac61e3245fdc0bb622578cdd651f7b15a039a8ac0"} Oct 30 13:16:09.995294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:09.995170 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:10.374620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:10.367199 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:10.593269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:10.593234 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:16:10.593662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:10.593290 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:16:11.260273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:11.260094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:11.594238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:11.594150 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:16:11.594238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:11.594210 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:16:12.162399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:12.162371 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:12.951170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:12.951135 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:12.951657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:12.951555 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:12.952395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:12.952355 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:16:13.005753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:13.005695 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" podStartSLOduration=4.005677734 podStartE2EDuration="4.005677734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:10.050831462 +0000 UTC m=+745.917449636" watchObservedRunningTime="2026-10-30 13:16:13.005677734 +0000 UTC m=+748.872295915" Oct 30 13:16:13.822107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:13.822067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:15.366804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:15.366774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:15.701040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:15.700935 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:17.119715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:17.119222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:19.191402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:19.191353 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:20.809803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:20.809761 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zklq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:16:20.810209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:20.809843 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:20.877665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:20.877587 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh"] Oct 30 13:16:20.883180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:20.883153 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:20.906074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:20.905233 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh"] Oct 30 13:16:20.956401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:20.956363 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000347 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v22\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000445 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000479 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000552 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000649 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000680 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000719 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.005622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.000762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.010807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.010777 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:21.102337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.102242 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.102500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.102365 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.102500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.102415 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.102500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.102456 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.103554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.102504 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.103554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.102701 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.103554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.103173 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.103554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.103457 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.104493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.104316 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.104579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.104556 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.105168 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.105117 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-132ab67347e8: secret "agc-cred-132ab67347e8" not found Oct 30 13:16:21.105168 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.105142 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1856bd8d477d: secret "agc-cred-1856bd8d477d" not found Oct 30 13:16:21.105168 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.105157 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9: [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:21.105380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.105357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v22\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.105460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.105418 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.105527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.105456 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.105699 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.105681 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials podName:2ace2e92-51cc-4bee-b050-b8138b997975 nodeName:}" failed. No retries permitted until 2026-10-30 13:16:21.605656936 +0000 UTC m=+757.472275087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials") pod "payload-processing-external-model-55d4cbbbf5-hg9m9" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975") : [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:21.106649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.106626 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.107884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.107860 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.108666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.108642 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.126937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.126906 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.127174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.126380 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v22\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.176202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.176166 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9"] Oct 30 13:16:21.218209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.218171 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:21.250262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.250233 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:21.390620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.390525 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:21.614370 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.614332 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-132ab67347e8: secret "agc-cred-132ab67347e8" not found Oct 30 13:16:21.614639 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.614627 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1856bd8d477d: secret "agc-cred-1856bd8d477d" not found Oct 30 13:16:21.614747 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.614738 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9: [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:21.614922 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:21.614909 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials podName:2ace2e92-51cc-4bee-b050-b8138b997975 nodeName:}" failed. No retries permitted until 2026-10-30 13:16:22.614871253 +0000 UTC m=+758.481489410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials") pod "payload-processing-external-model-55d4cbbbf5-hg9m9" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975") : [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:21.628663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.626080 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh"] Oct 30 13:16:21.640987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.640071 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:21.640987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.640165 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:21.640987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:21.640207 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:22.074251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:22.074210 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" event={"ID":"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93","Type":"ContainerStarted","Data":"4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978"} Oct 30 13:16:22.074645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:22.074287 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" event={"ID":"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93","Type":"ContainerStarted","Data":"db556b45afa4722ad0787caf5f036b5a2fad9eb4c4d51a961f6719445fe23bed"} Oct 30 13:16:22.219335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:22.219306 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:22.221137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:22.221111 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:22.221259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:22.221162 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:22.625933 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:22.625902 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-132ab67347e8: secret "agc-cred-132ab67347e8" not found Oct 30 13:16:22.625933 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:22.625928 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1856bd8d477d: secret "agc-cred-1856bd8d477d" not found Oct 30 13:16:22.625933 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:22.625938 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9: [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:22.626291 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:22.625985 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials podName:2ace2e92-51cc-4bee-b050-b8138b997975 nodeName:}" failed. No retries permitted until 2026-10-30 13:16:24.625971975 +0000 UTC m=+760.492590126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials") pod "payload-processing-external-model-55d4cbbbf5-hg9m9" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975") : [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:22.707468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:22.707425 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:23.076073 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:23.076038 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:16:23.218979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:23.218944 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:23.219172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:23.219000 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:24.219683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:24.219651 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:24.220107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:24.219701 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:24.426035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:24.425999 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:24.676238 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:24.676206 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-132ab67347e8: secret "agc-cred-132ab67347e8" not found Oct 30 13:16:24.676238 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:24.676239 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1856bd8d477d: secret "agc-cred-1856bd8d477d" not found Oct 30 13:16:24.676448 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:24.676251 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9: [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:24.676448 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:24.676310 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials podName:2ace2e92-51cc-4bee-b050-b8138b997975 nodeName:}" failed. No retries permitted until 2026-10-30 13:16:28.676291129 +0000 UTC m=+764.542909295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials") pod "payload-processing-external-model-55d4cbbbf5-hg9m9" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975") : [secret "agc-cred-132ab67347e8" not found, secret "agc-cred-1856bd8d477d" not found] Oct 30 13:16:25.218872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:25.218839 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:25.219041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:25.218887 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:26.177453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:26.177405 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" podUID="2ace2e92-51cc-4bee-b050-b8138b997975" containerName="payload-processing" containerID="cri-o://90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3" gracePeriod=25 Oct 30 13:16:26.206726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:26.206690 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:26.221856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:26.221399 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:26.221856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:26.221459 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:27.010626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.000102 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:27.072651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.066855 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podStartSLOduration=7.066836728 podStartE2EDuration="7.066836728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:22.115792295 +0000 UTC m=+757.982410471" watchObservedRunningTime="2026-10-30 13:16:27.066836728 +0000 UTC m=+762.933454902" Oct 30 13:16:27.143366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.143261 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume\") pod \"2ace2e92-51cc-4bee-b050-b8138b997975\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " Oct 30 13:16:27.143716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.143693 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials\") pod \"2ace2e92-51cc-4bee-b050-b8138b997975\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " Oct 30 13:16:27.143896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.143882 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume\") pod \"2ace2e92-51cc-4bee-b050-b8138b997975\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " Oct 30 13:16:27.144061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.144042 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6\") pod \"2ace2e92-51cc-4bee-b050-b8138b997975\" (UID: \"2ace2e92-51cc-4bee-b050-b8138b997975\") " Oct 30 13:16:27.154122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.153855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume" pod "2ace2e92-51cc-4bee-b050-b8138b997975" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:27.155942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.154418 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume" pod "2ace2e92-51cc-4bee-b050-b8138b997975" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:27.249539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.227119 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:27.249539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.227180 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:27.310621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.259865 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:16:27.310621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.259901 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ace2e92-51cc-4bee-b050-b8138b997975-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:16:27.310621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.280138 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6" pod "2ace2e92-51cc-4bee-b050-b8138b997975" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975"). InnerVolumeSpecName "kube-api-access-dgtj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:27.312713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.312507 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials" pod "2ace2e92-51cc-4bee-b050-b8138b997975" (UID: "2ace2e92-51cc-4bee-b050-b8138b997975"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:27.317207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.317140 5693 generic.go:356] "Generic (PLEG): container finished" podID="2ace2e92-51cc-4bee-b050-b8138b997975" containerID="90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3" exitCode=0 Oct 30 13:16:27.317475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.317441 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" event={"ID":"2ace2e92-51cc-4bee-b050-b8138b997975","Type":"ContainerDied","Data":"90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3"} Oct 30 13:16:27.317536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.317522 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" event={"ID":"2ace2e92-51cc-4bee-b050-b8138b997975","Type":"ContainerDied","Data":"6064088b565e2c298ae65a1ac61e3245fdc0bb622578cdd651f7b15a039a8ac0"} Oct 30 13:16:27.317592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.317559 5693 scope.go:122] "RemoveContainer" containerID="90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3" Oct 30 13:16:27.317827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.317799 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9" Oct 30 13:16:27.373848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.361122 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtj6\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-kube-api-access-dgtj6\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:16:27.373848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.361150 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ace2e92-51cc-4bee-b050-b8138b997975-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:16:27.406898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.406867 5693 scope.go:122] "RemoveContainer" containerID="90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3" Oct 30 13:16:27.407294 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:27.407266 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3\": container with ID starting with 90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3 not found: ID does not exist" containerID="90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3" Oct 30 13:16:27.407407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.407304 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3"} err="failed to get container status \"90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3\": rpc error: code = NotFound desc = could not find container \"90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3\": container with ID starting with 90ba8fb9c1d4916282f5a5be8099609d218ca0d9b1762d3dc0e85ce3b6f6e5d3 not found: ID does not exist" Oct 30 13:16:27.414414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.414389 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9"] Oct 30 13:16:27.416103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:27.416086 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55d4cbbbf5-hg9m9"] Oct 30 13:16:28.218897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:28.218864 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 13:16:28.219150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:28.218923 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 13:16:28.255011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:28.254973 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ace2e92-51cc-4bee-b050-b8138b997975" path="/var/lib/kubelet/pods/2ace2e92-51cc-4bee-b050-b8138b997975/volumes" Oct 30 13:16:28.342833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:28.342799 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:28.513622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:28.500211 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:30.044620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:30.044129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:30.261742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:30.261707 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:16:30.261928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:30.261771 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:30.793350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:30.793314 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:30.793774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:30.793744 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:30.794662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:30.794641 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:16:31.734654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:31.732977 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:33.313306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:33.313234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:35.032327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.032283 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:35.930630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.930570 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9"] Oct 30 13:16:35.931202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.931186 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2ace2e92-51cc-4bee-b050-b8138b997975" containerName="payload-processing" Oct 30 13:16:35.931277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.931207 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ace2e92-51cc-4bee-b050-b8138b997975" containerName="payload-processing" Oct 30 13:16:35.931375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.931364 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ace2e92-51cc-4bee-b050-b8138b997975" containerName="payload-processing" Oct 30 13:16:35.934740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.934725 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.981867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.981831 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982034 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982112 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982157 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982200 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982236 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982259 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982293 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.982378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.982322 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqzm\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:35.993581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:35.993527 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9"] Oct 30 13:16:36.083285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083249 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083293 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083414 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083454 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083474 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqzm\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083550 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083706 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.083805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083793 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.084183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083831 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.084183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083835 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.084183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.083907 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.084287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.084178 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.084287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.084220 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.084376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.084359 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.085728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.085710 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.085949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.085931 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.111203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.111181 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqzm\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.123984 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.123959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.251775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.251742 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:36.577419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.577254 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9"] Oct 30 13:16:36.584177 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:16:36.584140 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055c3603_f114_4fd7_8201_263f8d428598.slice/crio-2bbdee490fd4d41548fca5a46fb63675afe604c4dae609fe6d083f742022c6c1 WatchSource:0}: Error finding container 2bbdee490fd4d41548fca5a46fb63675afe604c4dae609fe6d083f742022c6c1: Status 404 returned error can't find the container with id 2bbdee490fd4d41548fca5a46fb63675afe604c4dae609fe6d083f742022c6c1 Oct 30 13:16:36.586572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.586527 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:36.586726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.586642 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:36.586726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:36.586692 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:16:37.022621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:37.018343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:37.535622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:37.535122 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" event={"ID":"055c3603-f114-4fd7-8201-263f8d428598","Type":"ContainerStarted","Data":"a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e"} Oct 30 13:16:37.535622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:37.535192 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" event={"ID":"055c3603-f114-4fd7-8201-263f8d428598","Type":"ContainerStarted","Data":"2bbdee490fd4d41548fca5a46fb63675afe604c4dae609fe6d083f742022c6c1"} Oct 30 13:16:38.312621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:38.303981 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:38.348982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:38.341929 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:38.348982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:38.341994 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:39.252262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:39.252224 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:39.252788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:39.252282 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:40.272533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:40.272487 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:40.412246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:40.412204 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:40.609674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:40.600853 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:41.278617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.269272 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:41.278617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.269344 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:41.500934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.500873 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podStartSLOduration=6.500853021 podStartE2EDuration="6.500853021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:37.617932196 +0000 UTC m=+773.484550403" watchObservedRunningTime="2026-10-30 13:16:41.500853021 +0000 UTC m=+777.367471196" Oct 30 13:16:41.504509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.504480 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk"] Oct 30 13:16:41.516680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.516653 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.545671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.545331 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk"] Oct 30 13:16:41.546912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.546532 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"distinct-w2-ec41e5-kserve-self-signed-certs\"" Oct 30 13:16:41.605553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.605449 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-685gd\" (UniqueName: \"kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.605553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.605538 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.605848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.605671 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.605848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.605762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.605848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.605793 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.605848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.605823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.615743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.615714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.723783 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-685gd\" (UniqueName: \"kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.723870 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.723964 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.724050 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.724076 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.724101 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.738621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.730418 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.762995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.745229 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.762995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.745508 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.762995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.745748 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.762995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.745976 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.832555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.822358 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-685gd\" (UniqueName: \"kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd\") pod \"distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:41.911675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:41.911636 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:42.257009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.256973 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk"] Oct 30 13:16:42.259509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.258688 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:42.259509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.258739 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:42.263913 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:16:42.263882 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4b78d06_b249_4247_b0e7_78acefe36d87.slice/crio-58d72d95543e1e974297daad31b57a71af5870c1b685976ecb45ac3e87cb3455 WatchSource:0}: Error finding container 58d72d95543e1e974297daad31b57a71af5870c1b685976ecb45ac3e87cb3455: Status 404 returned error can't find the container with id 58d72d95543e1e974297daad31b57a71af5870c1b685976ecb45ac3e87cb3455 Oct 30 13:16:42.653305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.653198 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:42.754777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.754735 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" event={"ID":"f4b78d06-b249-4247-b0e7-78acefe36d87","Type":"ContainerStarted","Data":"9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da"} Oct 30 13:16:42.754961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.754791 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" event={"ID":"f4b78d06-b249-4247-b0e7-78acefe36d87","Type":"ContainerStarted","Data":"58d72d95543e1e974297daad31b57a71af5870c1b685976ecb45ac3e87cb3455"} Oct 30 13:16:42.794322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.794262 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podStartSLOduration=1.794244382 podStartE2EDuration="1.794244382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:42.792448442 +0000 UTC m=+778.659066615" watchObservedRunningTime="2026-10-30 13:16:42.794244382 +0000 UTC m=+778.660862557" Oct 30 13:16:42.921722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:42.917695 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:43.254411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:43.254378 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:43.254685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:43.254423 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:44.253204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:44.252774 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:44.253850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:44.252861 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:44.487393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:44.487346 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:45.252900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:45.252866 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:16:45.253066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:45.252926 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:16:46.224702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:46.224654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:46.295237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:46.295192 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:46.296146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:46.296121 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:46.296829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:46.296803 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:16:47.900716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:47.900680 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:49.602988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:49.602949 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:50.726286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:50.726244 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:51.055765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:51.055691 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:51.222857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:51.222797 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:51.912393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:51.912362 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:51.924973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:51.924947 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:52.315940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:52.315912 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:52.826286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:52.826259 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:16:52.905481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:52.905450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:53.064832 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:16:53.064790 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:16:54.805749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:54.805709 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:56.455083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.455046 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm"] Oct 30 13:16:56.461204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.461179 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.480063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.480034 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm"] Oct 30 13:16:56.487986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.487895 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"openshift-service-ca.crt\"" Oct 30 13:16:56.487986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.487928 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"default-dockercfg-g8v59\"" Oct 30 13:16:56.487986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.487987 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"distinct-2-w0-f9ebb5-kserve-self-signed-certs\"" Oct 30 13:16:56.487986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.487994 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"kube-root-ca.crt\"" Oct 30 13:16:56.566733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.566692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:56.619386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.619350 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.619548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.619395 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.619548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.619520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkrk\" (UniqueName: \"kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.619675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.619589 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.619675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.619642 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.619750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.619694 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.728586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.727715 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.728586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.727784 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.728586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.727879 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkrk\" (UniqueName: \"kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.728586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.728465 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.728586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.728516 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.728586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.728556 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.729629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.729572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.729862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.729841 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.730081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.730062 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.734747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.731957 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.734747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.733558 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.736128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.736085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:56.765629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.765383 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkrk\" (UniqueName: \"kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk\") pod \"distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:56.786575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:56.786535 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:16:57.099775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:57.099753 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm"] Oct 30 13:16:57.102157 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:16:57.102113 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cffe0f8_7282_4e8f_ae05_d15c86ef35d4.slice/crio-dbda0be3b5e3c0fbe34d31873bf28df926d48703e0aef2905880fa5b7cecd468 WatchSource:0}: Error finding container dbda0be3b5e3c0fbe34d31873bf28df926d48703e0aef2905880fa5b7cecd468: Status 404 returned error can't find the container with id dbda0be3b5e3c0fbe34d31873bf28df926d48703e0aef2905880fa5b7cecd468 Oct 30 13:16:57.919358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:57.919325 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" event={"ID":"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4","Type":"ContainerStarted","Data":"1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e"} Oct 30 13:16:57.919807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:57.919369 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" event={"ID":"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4","Type":"ContainerStarted","Data":"dbda0be3b5e3c0fbe34d31873bf28df926d48703e0aef2905880fa5b7cecd468"} Oct 30 13:16:57.960649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:57.960579 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podStartSLOduration=1.9605646559999999 podStartE2EDuration="1.960564656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:57.957523539 +0000 UTC m=+793.824141712" watchObservedRunningTime="2026-10-30 13:16:57.960564656 +0000 UTC m=+793.827182828" Oct 30 13:16:58.534474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:58.534351 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:58.787489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:16:58.787408 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:17:00.216324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:00.216276 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:01.903879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:01.903844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:03.532556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:03.532514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:05.108779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.108737 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:05.457316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.457237 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg"] Oct 30 13:17:05.461321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.461302 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.496947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.490837 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg"] Oct 30 13:17:05.564916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.564881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.564940 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565017 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565067 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565127 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565200 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565285 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchhw\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565326 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.565422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.565411 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666221 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666276 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666330 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666362 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666430 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666461 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666587 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666702 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchhw\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666731 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666770 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.666829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666801 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.667030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666853 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.667030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.666992 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.667224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.667202 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.668704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.668681 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.668826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.668810 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.684664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.684641 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.685062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.685041 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchhw\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:05.777678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:05.777650 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:06.093189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.093162 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg"] Oct 30 13:17:06.095683 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:17:06.095654 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda735ba9b_70f3_4ea2_ae2d_22677e168c19.slice/crio-89a3f1764813d18cf1f6340073fe4c13fada23d3538869fb6560af75395dc99c WatchSource:0}: Error finding container 89a3f1764813d18cf1f6340073fe4c13fada23d3538869fb6560af75395dc99c: Status 404 returned error can't find the container with id 89a3f1764813d18cf1f6340073fe4c13fada23d3538869fb6560af75395dc99c Oct 30 13:17:06.098053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.098018 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:06.098142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.098086 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:06.098142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.098118 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:06.316983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.316940 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:06.787289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.787252 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:17:06.800246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.800220 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:17:06.923982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:06.923835 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:07.004775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.004728 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" event={"ID":"a735ba9b-70f3-4ea2-ae2d-22677e168c19","Type":"ContainerStarted","Data":"c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8"} Oct 30 13:17:07.004987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.004788 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" event={"ID":"a735ba9b-70f3-4ea2-ae2d-22677e168c19","Type":"ContainerStarted","Data":"89a3f1764813d18cf1f6340073fe4c13fada23d3538869fb6560af75395dc99c"} Oct 30 13:17:07.016571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.016545 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:17:07.048685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.047586 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podStartSLOduration=2.047567329 podStartE2EDuration="2.047567329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:07.046118674 +0000 UTC m=+802.912736861" watchObservedRunningTime="2026-10-30 13:17:07.047567329 +0000 UTC m=+802.914185503" Oct 30 13:17:07.777892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.777860 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:07.779513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.779488 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:07.779650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:07.779536 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:08.467621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.466663 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7"] Oct 30 13:17:08.617764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.617728 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7"] Oct 30 13:17:08.618118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.618047 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.712078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.712043 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.712467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.712444 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.712728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.712706 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.712998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.712968 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.713223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.713205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.713623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.713558 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.713775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.713757 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.713963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.713948 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.714110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.714094 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79k5\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.787491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.787036 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:08.787491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.787101 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820073 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820129 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820205 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820240 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820269 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820302 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820402 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820463 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79k5\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.820530 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.822223 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.822531 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.822840 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.823625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.823617 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.833621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.824555 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.854636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.850753 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.858627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.856795 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79k5\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.870120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.867079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:08.870561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:08.870536 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:09.021808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.018859 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:09.220212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.220109 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:09.732624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.728241 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7"] Oct 30 13:17:09.773944 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:17:09.773888 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedafbadc_9d8c_4547_ba9d_b721c3b254a5.slice/crio-5d998f6f068a84125c80188632991f14b2cf78dada452d5d5ba06ffe42cd67b0 WatchSource:0}: Error finding container 5d998f6f068a84125c80188632991f14b2cf78dada452d5d5ba06ffe42cd67b0: Status 404 returned error can't find the container with id 5d998f6f068a84125c80188632991f14b2cf78dada452d5d5ba06ffe42cd67b0 Oct 30 13:17:09.790452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.790001 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:09.790452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.790062 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:09.804538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.803827 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:09.804538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.803970 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:09.804538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:09.804048 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:10.423622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:10.420450 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" event={"ID":"edafbadc-9d8c-4547-ba9d-b721c3b254a5","Type":"ContainerStarted","Data":"5d998f6f068a84125c80188632991f14b2cf78dada452d5d5ba06ffe42cd67b0"} Oct 30 13:17:10.786223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:10.785764 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:10.786223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:10.785826 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:10.818136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:10.818088 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:10.928620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:10.923743 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:11.028674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.028380 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:17:11.343055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.342957 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" event={"ID":"edafbadc-9d8c-4547-ba9d-b721c3b254a5","Type":"ContainerStarted","Data":"5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b"} Oct 30 13:17:11.454996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.454927 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podStartSLOduration=3.454904081 podStartE2EDuration="3.454904081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:11.417249181 +0000 UTC m=+807.283867355" watchObservedRunningTime="2026-10-30 13:17:11.454904081 +0000 UTC m=+807.321522255" Oct 30 13:17:11.779619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.779004 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:11.779619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.779070 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:11.887279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.887188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:11.930104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.927725 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:11.930104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:11.928537 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:17:12.021855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:12.021768 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:12.035683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:12.028527 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:12.035683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:12.028587 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:12.219617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:12.219565 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:17:12.778835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:12.778797 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:12.779159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:12.779106 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:13.019793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:13.019756 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:13.020195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:13.019814 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:13.778507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:13.778472 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:17:13.778708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:13.778520 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:17:13.862015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:13.861972 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:14.020488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:14.020449 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:14.020943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:14.020509 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:15.020142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:15.020099 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:15.020342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:15.020169 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:15.157958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:15.157910 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:15.158564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:15.158537 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:15.159856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:15.159823 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:17:15.651168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:15.651117 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:16.020366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:16.020331 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:16.020554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:16.020386 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:17.020337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:17.020305 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:17.020747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:17.020354 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:17.505383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:17.505352 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:18.020369 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:18.020335 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:17:18.020788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:18.020399 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:17:19.024208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:19.024177 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:19.024574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:19.024435 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:19.025146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:19.025130 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:17:19.300360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:19.294728 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:21.213624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:21.212674 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:22.680760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:22.680714 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:22.753180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:22.752155 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:22.920758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:22.920709 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:23.153382 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:17:23.153345 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:17:23.704150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:23.704108 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:24.641795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:24.641755 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:25.426072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:25.425996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:27.216370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:27.216337 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:28.946709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:28.946671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:29.366521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:29.366485 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:30.923901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:30.923868 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:32.210332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:32.210294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:33.921057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:33.921019 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:35.503291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:35.503242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:37.223283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:37.223241 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:38.177153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.177120 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d"] Oct 30 13:17:38.182699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.182671 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.202368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.202337 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d"] Oct 30 13:17:38.470671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.439477 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ct79\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.514763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.514721 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.515136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.515108 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.526775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.526730 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.531621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.527130 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.621006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.620963 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.621331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.621308 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.645732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.645689 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.646094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.646072 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.750452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.750413 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.750829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.750806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.750985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.750966 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.751099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.751084 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.751269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.751253 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.751448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.751431 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ct79\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.751585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.751570 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.751753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.751734 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.751909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.751893 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.752888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.752861 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.753292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.753269 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.755048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.753805 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.758674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.754228 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.758808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.757420 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.759018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.758996 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.779119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.779069 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.783625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.783584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ct79\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.814516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.814489 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:38.834863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:38.834832 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:40.432124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:40.432072 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:40.432754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:40.432722 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:40.432936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:40.432913 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:40.516970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:40.516888 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d"] Oct 30 13:17:40.823029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:40.822716 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:41.169700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:41.169007 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:41.311042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:41.310999 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" event={"ID":"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6","Type":"ContainerStarted","Data":"620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b"} Oct 30 13:17:41.311357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:41.311322 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" event={"ID":"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6","Type":"ContainerStarted","Data":"00e732e3543b01fc4255dd7733dfea6134d41ae19aa671c19a5ef44688e91bcf"} Oct 30 13:17:41.836398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:41.836356 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:41.837552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:41.837530 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:41.837622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:41.837584 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:42.851109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:42.847738 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:42.851109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:42.847808 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:42.953707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:42.953666 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:43.090418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.090356 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podStartSLOduration=5.090341126 podStartE2EDuration="5.090341126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:41.407440483 +0000 UTC m=+837.274058657" watchObservedRunningTime="2026-10-30 13:17:43.090341126 +0000 UTC m=+838.956959299" Oct 30 13:17:43.091339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.091305 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv"] Oct 30 13:17:43.095694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.095672 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.121821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.121793 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv"] Oct 30 13:17:43.183990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.183952 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.183990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.183997 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184089 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184196 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184302 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184340 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184364 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184440 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpzc\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.184560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.184480 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.285790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.285755 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.285972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.285806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.285972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.285829 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.285999 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpzc\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286149 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286219 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286245 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286328 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286688 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286734 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286796 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.286941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.286829 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.287058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.287030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.287637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.287591 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.288869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.288842 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.289219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.289196 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.308445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.308412 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.308567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.308495 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpzc\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.409857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.409771 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:43.756967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.756935 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv"] Oct 30 13:17:43.757592 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:17:43.757565 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96219367_d332_4190_b714_4d319878b0cb.slice/crio-d4eb61259562af64aa93f3aa3fb4a94dda67d5decd1e1cff0236368593de7efc WatchSource:0}: Error finding container d4eb61259562af64aa93f3aa3fb4a94dda67d5decd1e1cff0236368593de7efc: Status 404 returned error can't find the container with id d4eb61259562af64aa93f3aa3fb4a94dda67d5decd1e1cff0236368593de7efc Oct 30 13:17:43.759991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.759958 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:43.760077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.760031 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:43.760077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.760061 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:17:43.849695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.849568 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:43.849695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:43.849640 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:44.392862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.392827 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" event={"ID":"96219367-d332-4190-b714-4d319878b0cb","Type":"ContainerStarted","Data":"3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e"} Oct 30 13:17:44.393380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.392901 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" event={"ID":"96219367-d332-4190-b714-4d319878b0cb","Type":"ContainerStarted","Data":"d4eb61259562af64aa93f3aa3fb4a94dda67d5decd1e1cff0236368593de7efc"} Oct 30 13:17:44.410162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.410132 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:44.411844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.411818 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:44.411982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.411860 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:44.483162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.483097 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podStartSLOduration=1.4830761909999999 podStartE2EDuration="1.483076191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:44.449288621 +0000 UTC m=+840.315906795" watchObservedRunningTime="2026-10-30 13:17:44.483076191 +0000 UTC m=+840.349694365" Oct 30 13:17:44.823964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.823928 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:44.842622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.842017 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:44.842622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:44.842077 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:45.411017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:45.410982 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:45.411414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:45.411062 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:45.846623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:45.836958 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:45.846623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:45.837023 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:46.410668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:46.410510 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:46.410668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:46.410580 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:46.845625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:46.842979 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:46.845625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:46.843040 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:46.909068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:46.909037 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:47.090072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:47.090041 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:47.410206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:47.410172 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:47.410461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:47.410434 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:47.844137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:47.844094 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:17:47.844427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:47.844396 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 13:17:48.410265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:48.410233 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:48.410692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:48.410283 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:48.851626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:48.851197 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:49.410769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:49.410739 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:49.411231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:49.411201 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:49.416075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:49.416051 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:49.416592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:49.416571 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:49.417634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:49.417582 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:17:50.410337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:50.410302 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 13:17:50.410540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:50.410376 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 13:17:50.651131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:50.651089 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:51.414529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:51.414501 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:51.414822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:51.414800 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:51.415525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:51.415508 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:17:52.329810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:52.329774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:52.476633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:52.473849 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:53.055756 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:17:53.055711 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:17:55.035356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:55.035318 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:55.742325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:55.742287 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:57.607624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:57.606706 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:17:57.607624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:57.606812 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:57.795620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:57.794693 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:58.009627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:58.006865 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:58.246663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:58.245693 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:58.770623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:58.763019 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:58.945629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:58.941244 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:59.036418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.034074 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:17:59.036418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.034674 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:17:59.343679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.337572 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:59.343679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.338204 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:17:59.357887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.357259 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:17:59.357887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.357375 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:59.697743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.696615 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:17:59.749521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.749230 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:17:59.858617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:17:59.858565 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:01.637145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:01.637024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:03.411829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:03.411795 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:05.002390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:05.002354 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:06.705011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:06.704153 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:08.305394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.305354 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:08.475556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.475525 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf"] Oct 30 13:18:08.480335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.480313 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.497365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.497338 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf"] Oct 30 13:18:08.505614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.505566 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"distinct-2-w4-9bf8ad-kserve-self-signed-certs\"" Oct 30 13:18:08.505748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.505726 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"default-dockercfg-4lf5d\"" Oct 30 13:18:08.506847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.506820 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"kube-root-ca.crt\"" Oct 30 13:18:08.506967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.506906 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"openshift-service-ca.crt\"" Oct 30 13:18:08.573153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.573074 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:08.585977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.585941 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.586168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.586126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.586245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.586166 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hfk\" (UniqueName: \"kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.586303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.586262 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.586359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.586318 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.586439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.586424 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.690458 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.690570 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.695800 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.695865 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfk\" (UniqueName: \"kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.695927 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.696002 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.696536 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.696808 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.697734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.697461 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.706155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.706125 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.706516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.706489 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.735524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.735475 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hfk\" (UniqueName: \"kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk\") pod \"distinct-2-w4-9bf8ad-kserve-794785859d-4szsf\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:08.801148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:08.801110 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:09.109756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.109734 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf"] Oct 30 13:18:09.112965 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:09.112902 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4529cf2d_afc8_4338_b21e_65bdf6cdf76e.slice/crio-1a8e26c896f95f90ec47abd2aeb71c30766e08929a80997cd1afb4a2e49c22da WatchSource:0}: Error finding container 1a8e26c896f95f90ec47abd2aeb71c30766e08929a80997cd1afb4a2e49c22da: Status 404 returned error can't find the container with id 1a8e26c896f95f90ec47abd2aeb71c30766e08929a80997cd1afb4a2e49c22da Oct 30 13:18:09.713760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.712254 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:09.798941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.798908 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" event={"ID":"4529cf2d-afc8-4338-b21e-65bdf6cdf76e","Type":"ContainerStarted","Data":"b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed"} Oct 30 13:18:09.799098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.798950 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" event={"ID":"4529cf2d-afc8-4338-b21e-65bdf6cdf76e","Type":"ContainerStarted","Data":"1a8e26c896f95f90ec47abd2aeb71c30766e08929a80997cd1afb4a2e49c22da"} Oct 30 13:18:09.802199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.802179 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:09.840907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.840859 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podStartSLOduration=1.84084635 podStartE2EDuration="1.84084635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:09.837360704 +0000 UTC m=+865.703978878" watchObservedRunningTime="2026-10-30 13:18:09.84084635 +0000 UTC m=+865.707464523" Oct 30 13:18:09.878873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:09.878836 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:10.066191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:10.051067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:11.709472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:11.709442 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:13.247847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.247815 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq"] Oct 30 13:18:13.252540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.252517 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.276997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.276971 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq"] Oct 30 13:18:13.311633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.311564 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364498 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6sp\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364583 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364635 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364686 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364772 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364837 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364874 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.364917 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.365289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.365053 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466024 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466074 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466114 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466221 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466301 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6sp\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466352 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466380 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466423 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466503 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466517 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466544 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466613 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466841 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.466938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.466903 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.468806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.468767 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.469220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.469197 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.484860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.484837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6sp\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.500863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.500794 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.569386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.569356 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:13.893630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.893577 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq"] Oct 30 13:18:13.895630 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:13.895588 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860ef2b5_d62c_4059_90e9_e59a2019141c.slice/crio-25e247f9968fd11083869dd0eabc85846b5357326dcf7f04819537c375f22f97 WatchSource:0}: Error finding container 25e247f9968fd11083869dd0eabc85846b5357326dcf7f04819537c375f22f97: Status 404 returned error can't find the container with id 25e247f9968fd11083869dd0eabc85846b5357326dcf7f04819537c375f22f97 Oct 30 13:18:13.897888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.897851 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:13.897980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.897914 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:13.897980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:13.897961 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:14.858223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:14.858179 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" event={"ID":"860ef2b5-d62c-4059-90e9-e59a2019141c","Type":"ContainerStarted","Data":"185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743"} Oct 30 13:18:14.858680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:14.858245 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" event={"ID":"860ef2b5-d62c-4059-90e9-e59a2019141c","Type":"ContainerStarted","Data":"25e247f9968fd11083869dd0eabc85846b5357326dcf7f04819537c375f22f97"} Oct 30 13:18:14.924496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:14.924436 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podStartSLOduration=1.924422053 podStartE2EDuration="1.924422053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:14.909556229 +0000 UTC m=+870.776174401" watchObservedRunningTime="2026-10-30 13:18:14.924422053 +0000 UTC m=+870.791040226" Oct 30 13:18:15.039673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:15.039635 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:15.569762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:15.569724 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:15.571415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:15.571383 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:15.571538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:15.571437 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:16.569948 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:16.569913 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:16.570456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:16.569962 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:16.762633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:16.759379 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:17.049630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:17.048438 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:17.224621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:17.213272 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:17.403699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:17.392263 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:17.581622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:17.577114 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:17.581622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:17.577205 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:18.259628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.258143 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:18.466619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.456158 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:18.575534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.575017 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:18.575534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.575083 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:18.635024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.634991 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:18.711661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.706690 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:18.792622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.777806 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:18.792622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.778987 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:18.819001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.818966 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:18.822089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:18.822054 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:19.573623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.572298 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:19.573623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.572360 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:19.721666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.720705 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:19.721666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.721510 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:18:19.781625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.779200 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:19.781625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.779580 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:19.781625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.780204 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:18:19.781625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.780261 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:18:19.819028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.818926 5693 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:19.919705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.848738 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 13:18:19.919705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:19.849183 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:18:20.543419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.531446 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:18:20.543419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.531528 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:20.576189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.576147 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:20.580621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.577691 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:20.580621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.577751 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:20.722960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.721689 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:20.782775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.782721 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:20.784738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.784684 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:18:20.850834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:20.850728 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:21.575619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.575138 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:21.575619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.575249 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:21.797671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.735508 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:18:21.824623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.801385 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:18:21.826455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.805184 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:18:21.827810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.805550 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:21.880799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.880745 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:21.882121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:21.882020 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:18:22.499218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.494417 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:18:22.499218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.494488 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:22.518005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.517960 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:22.592892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.589363 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:22.592892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.589523 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:22.848659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.848520 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:22.882781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.882736 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:22.924640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:22.924562 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 13:18:23.018985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.018938 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:18:23.115938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.115854 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:18:23.162039 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:18:23.161976 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:18:23.320682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.320645 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9"] Oct 30 13:18:23.326393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.326365 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.376777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.374705 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72"] Oct 30 13:18:23.380451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.380025 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.402489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.402458 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9"] Oct 30 13:18:23.409392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.409365 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72"] Oct 30 13:18:23.422458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422427 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.422585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422481 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.422693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422579 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cl5\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.422693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422649 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.422803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422717 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.422803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422742 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.422803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422788 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.423012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.422989 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.423065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423042 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.423239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423219 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423282 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423325 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.423429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423412 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7b7\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423447 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423511 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423625 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.423728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.423711 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.524981 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525019 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525044 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525067 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cl5\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525097 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525140 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525163 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525206 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525277 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525323 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525374 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525447 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525483 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525507 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525556 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7b7\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525561 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525590 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525632 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525703 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.525752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525703 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.526387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.525922 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.526387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.526050 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.526387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.526123 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.526387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.526237 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.526387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.526352 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.526666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.526393 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.526666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.526491 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.548658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.547002 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.550915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.549501 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.550915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.549976 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.550915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.550196 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.550915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.550612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.551280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.551012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.552232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.552207 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cl5\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.552345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.552205 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7b7\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:23.570350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.570310 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:23.570513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.570377 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:23.700468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.700365 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:23.708799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:23.708769 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:24.076658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.076627 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72"] Oct 30 13:18:24.079420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.079386 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9"] Oct 30 13:18:24.091949 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:24.091915 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194f40db_5435_4052_982e_5156e160cd7e.slice/crio-eeddd5f78d832ab81828296f95923722354ce9d4426937f2ff3b2e061532a6ef WatchSource:0}: Error finding container eeddd5f78d832ab81828296f95923722354ce9d4426937f2ff3b2e061532a6ef: Status 404 returned error can't find the container with id eeddd5f78d832ab81828296f95923722354ce9d4426937f2ff3b2e061532a6ef Oct 30 13:18:24.093011 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:24.092924 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1ab878_6a5a_4513_a340_9b3ea31b7d61.slice/crio-15699e58db8848bdae4deb05a1428b5911a0ea3d4d94c1203f8bcdf2fd58e4d0 WatchSource:0}: Error finding container 15699e58db8848bdae4deb05a1428b5911a0ea3d4d94c1203f8bcdf2fd58e4d0: Status 404 returned error can't find the container with id 15699e58db8848bdae4deb05a1428b5911a0ea3d4d94c1203f8bcdf2fd58e4d0 Oct 30 13:18:24.095242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.095204 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:24.095324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.095293 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:24.095382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.095333 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:24.095419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.095396 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:24.095468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.095454 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:24.095500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.095491 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:24.241642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.240398 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:24.374488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.374396 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:24.540367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.540331 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:24.582620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.572961 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:18:24.582620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.573022 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:18:24.635924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.635838 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" event={"ID":"194f40db-5435-4052-982e-5156e160cd7e","Type":"ContainerStarted","Data":"521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783"} Oct 30 13:18:24.635924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.635911 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" event={"ID":"194f40db-5435-4052-982e-5156e160cd7e","Type":"ContainerStarted","Data":"eeddd5f78d832ab81828296f95923722354ce9d4426937f2ff3b2e061532a6ef"} Oct 30 13:18:24.639132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.639101 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" event={"ID":"9a1ab878-6a5a-4513-a340-9b3ea31b7d61","Type":"ContainerStarted","Data":"0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba"} Oct 30 13:18:24.639265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.639152 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" event={"ID":"9a1ab878-6a5a-4513-a340-9b3ea31b7d61","Type":"ContainerStarted","Data":"15699e58db8848bdae4deb05a1428b5911a0ea3d4d94c1203f8bcdf2fd58e4d0"} Oct 30 13:18:24.695989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.695934 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podStartSLOduration=1.695915995 podStartE2EDuration="1.695915995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:24.693139711 +0000 UTC m=+880.559757885" watchObservedRunningTime="2026-10-30 13:18:24.695915995 +0000 UTC m=+880.562534166" Oct 30 13:18:24.701218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.701192 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:24.709085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.709056 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:24.737112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.737065 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podStartSLOduration=1.737051447 podStartE2EDuration="1.737051447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:24.734766089 +0000 UTC m=+880.601384259" watchObservedRunningTime="2026-10-30 13:18:24.737051447 +0000 UTC m=+880.603669611" Oct 30 13:18:24.976620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:24.974876 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:25.354542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.354436 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n"] Oct 30 13:18:25.398623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.389157 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.413122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.413096 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n"] Oct 30 13:18:25.417065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.417038 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9fdad5-openshift-default-dockercfg-bkbbk\"" Oct 30 13:18:25.483681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.477058 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:25.513620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.513083 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.513620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.513158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.513620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.513198 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.513620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.513223 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvh5\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.521619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.515467 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.521619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.515509 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.560625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.524134 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.560625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.524178 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.560625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.524224 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626485 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626533 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvh5\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626591 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626708 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626907 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626931 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.626983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.626969 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.627190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.627099 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.627190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.627141 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.635643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.628349 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.635643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.629102 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.635643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.629761 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.635643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.630012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.636354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.636331 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.687384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.687290 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvh5\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.718622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.714251 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:25.718622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.714306 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:25.734622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.728713 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:25.734622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.728771 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:25.734622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.729710 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:25.750623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.739316 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.777348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.777230 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:25.778192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:25.778140 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo\") pod \"e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:26.014623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.010632 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:26.023006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.022971 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:26.506705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.503760 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n"] Oct 30 13:18:26.510280 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:26.510254 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61857bbf_18b2_464f_a4f5_2d949422a51d.slice/crio-78114f89c5f53637514f0f0904c869c22f268e14c71825bd3735dfd19cad3da0 WatchSource:0}: Error finding container 78114f89c5f53637514f0f0904c869c22f268e14c71825bd3735dfd19cad3da0: Status 404 returned error can't find the container with id 78114f89c5f53637514f0f0904c869c22f268e14c71825bd3735dfd19cad3da0 Oct 30 13:18:26.513926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.513877 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:26.514016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.513988 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:26.514075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.514040 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:26.615044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.614711 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:18:26.615044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.614797 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:26.719653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.717535 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:26.719653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.717623 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:26.734298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.733729 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:26.734298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.733807 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:26.734298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.734237 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:26.768691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:26.758849 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" event={"ID":"61857bbf-18b2-464f-a4f5-2d949422a51d","Type":"ContainerStarted","Data":"78114f89c5f53637514f0f0904c869c22f268e14c71825bd3735dfd19cad3da0"} Oct 30 13:18:27.399770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.399732 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:27.400433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.400408 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:27.401644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.401468 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:18:27.701108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.701028 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:27.701108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.701077 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:27.709410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.709382 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:27.709509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.709429 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:27.785945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.785903 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" event={"ID":"61857bbf-18b2-464f-a4f5-2d949422a51d","Type":"ContainerStarted","Data":"6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62"} Oct 30 13:18:27.851621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:27.850087 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podStartSLOduration=2.850066036 podStartE2EDuration="2.850066036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:27.825323178 +0000 UTC m=+883.691941352" watchObservedRunningTime="2026-10-30 13:18:27.850066036 +0000 UTC m=+883.716684211" Oct 30 13:18:28.026485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.026453 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:28.068970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.068936 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:28.701288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.701248 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:28.701779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.701304 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:28.709797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.709765 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:28.710040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.710012 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:28.797363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.796831 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:28.798140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.798113 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:18:28.815965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.815898 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:28.824210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.823343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:28.883760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:28.882163 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:29.298331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:29.298284 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:29.702210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:29.702119 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:29.702938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:29.702885 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:29.710227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:29.710205 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:29.710386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:29.710369 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:29.906086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:29.906055 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:30.187524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.187484 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:30.523451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.523416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:30.682702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.682660 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:30.701366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.701327 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:30.701516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.701397 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:30.709225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.709201 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:30.709617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:30.709254 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:31.107632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:31.106938 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:31.700866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:31.700831 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:18:31.701085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:31.700882 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:18:31.709166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:31.709139 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 13:18:31.709310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:31.709185 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 13:18:32.218906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.218759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:32.302158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.302123 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz"] Oct 30 13:18:32.307400 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.307367 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.309942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.309906 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz"] Oct 30 13:18:32.345957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.345926 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9fdad5\"" Oct 30 13:18:32.346130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.346003 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9fdad5-dockercfg-qkhzh\"" Oct 30 13:18:32.348320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.348298 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2"] Oct 30 13:18:32.354012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.353994 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.376125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.376035 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:32.378698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.378648 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2"] Oct 30 13:18:32.492409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.483036 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4npj\" (UniqueName: \"kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj\") pod \"payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.492409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.483184 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.492409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.483253 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.492409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.483399 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjdr\" (UniqueName: \"kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr\") pod \"payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.652491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.652444 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.652704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.652556 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.652784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.652734 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjdr\" (UniqueName: \"kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr\") pod \"payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.652877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.652857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4npj\" (UniqueName: \"kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj\") pod \"payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.654212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.654180 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.657050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.657020 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.740623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.738911 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4npj\" (UniqueName: \"kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj\") pod \"payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.751669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.746409 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:32.751669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.747480 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjdr\" (UniqueName: \"kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr\") pod \"payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.851411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.851364 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:32.861630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.855673 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:32.861630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.858084 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:18:32.931936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.931590 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:32.995341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.995309 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:32.996529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:32.996484 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:33.006498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.004189 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:18:33.163044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.162984 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2"] Oct 30 13:18:33.163949 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:33.163915 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b77ff48_6dde_4641_ac4d_62d7f46f4f8a.slice/crio-9afa15b6ed12b8892726921ca1695ae3c02e25f326e1a258bf37e6997d11afdb WatchSource:0}: Error finding container 9afa15b6ed12b8892726921ca1695ae3c02e25f326e1a258bf37e6997d11afdb: Status 404 returned error can't find the container with id 9afa15b6ed12b8892726921ca1695ae3c02e25f326e1a258bf37e6997d11afdb Oct 30 13:18:33.247542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.247509 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:33.284114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.284086 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz"] Oct 30 13:18:33.284952 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:33.284928 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39bbefe_f2dd_4f9c_8231_10e843fa5f3a.slice/crio-44e7ce44392f14260b3c0a4bbc2ee3202bffcadd0beb9b3b6e948a2a43a3face WatchSource:0}: Error finding container 44e7ce44392f14260b3c0a4bbc2ee3202bffcadd0beb9b3b6e948a2a43a3face: Status 404 returned error can't find the container with id 44e7ce44392f14260b3c0a4bbc2ee3202bffcadd0beb9b3b6e948a2a43a3face Oct 30 13:18:33.956929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.956893 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" event={"ID":"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a","Type":"ContainerStarted","Data":"c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0"} Oct 30 13:18:33.957419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.956955 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" event={"ID":"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a","Type":"ContainerStarted","Data":"44e7ce44392f14260b3c0a4bbc2ee3202bffcadd0beb9b3b6e948a2a43a3face"} Oct 30 13:18:33.957419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.956982 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:33.958639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.958612 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" event={"ID":"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a","Type":"ContainerStarted","Data":"5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e"} Oct 30 13:18:33.958772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.958660 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" event={"ID":"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a","Type":"ContainerStarted","Data":"9afa15b6ed12b8892726921ca1695ae3c02e25f326e1a258bf37e6997d11afdb"} Oct 30 13:18:33.996743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:33.996701 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" podStartSLOduration=1.99668949 podStartE2EDuration="1.99668949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:33.994054929 +0000 UTC m=+889.860673102" watchObservedRunningTime="2026-10-30 13:18:33.99668949 +0000 UTC m=+889.863307690" Oct 30 13:18:34.030810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:34.030764 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" podStartSLOduration=2.030748734 podStartE2EDuration="2.030748734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:34.030522516 +0000 UTC m=+889.897140689" watchObservedRunningTime="2026-10-30 13:18:34.030748734 +0000 UTC m=+889.897366921" Oct 30 13:18:34.243540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:34.237125 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.381810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:34.381772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.966570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:34.966539 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:35.293115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.288257 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.688506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.688425 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.781941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.781908 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z"] Oct 30 13:18:35.786747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.786723 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.798812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.798780 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw"] Oct 30 13:18:35.803389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.803354 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.814265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.812662 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z"] Oct 30 13:18:35.828533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.828504 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw"] Oct 30 13:18:35.833832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.833804 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.833981 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834026 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834060 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtczw\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834104 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834187 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834315 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834411 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834505 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834581 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834625 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834665 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834685 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834714 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.834767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834729 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834743 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.834767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.834761 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrnp\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.917647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.917582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.935550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935513 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.935709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935555 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.935709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.935709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935696 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.935874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935737 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.935874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935763 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.935874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935820 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.935874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935854 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935901 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935925 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935950 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.935983 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrnp\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936055 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936080 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936119 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936152 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtczw\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936197 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936299 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936574 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.936712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936664 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936874 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.936987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.936941 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.937057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.937029 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.937133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.937111 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.937279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.937263 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.937410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.937387 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.938033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.938006 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.938288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.938261 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.938586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.938545 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.939151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.939122 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.956316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.956295 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.956744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.956714 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtczw\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:35.956847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.956783 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrnp\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.956987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.956968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:35.973699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.973670 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm"] Oct 30 13:18:35.979572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.979553 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:35.995723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.995685 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b7b93a3\"" Oct 30 13:18:35.997241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:35.997208 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm"] Oct 30 13:18:36.037985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.037949 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.038261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.038229 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxj4\" (UniqueName: \"kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.038434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.038408 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b7b93a3-dockercfg-cx4q9\"" Oct 30 13:18:36.068814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.068785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:36.129622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.121874 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:36.166620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.150706 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxj4\" (UniqueName: \"kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.166620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.150923 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.166620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.151837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.166620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.152024 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:36.221672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.178500 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxj4\" (UniqueName: \"kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.348855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.344029 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:36.712741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.712715 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm"] Oct 30 13:18:36.715032 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:36.715003 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0e0cae_bd47_4c25_beaa_d66a9378541e.slice/crio-0ca1e71c00fa27b4cc60e29f150ccdb1310c8652c91ce7a993a00962bf317ddb WatchSource:0}: Error finding container 0ca1e71c00fa27b4cc60e29f150ccdb1310c8652c91ce7a993a00962bf317ddb: Status 404 returned error can't find the container with id 0ca1e71c00fa27b4cc60e29f150ccdb1310c8652c91ce7a993a00962bf317ddb Oct 30 13:18:36.716434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.716416 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z"] Oct 30 13:18:36.721150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.721121 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw"] Oct 30 13:18:36.732179 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:36.732145 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1516a792_85d5_46b7_b2e3_230efeb5fe70.slice/crio-9fa7e03a4300dc059db31355a8eb206204e0c9a9abe056220b07b84618fd5dc7 WatchSource:0}: Error finding container 9fa7e03a4300dc059db31355a8eb206204e0c9a9abe056220b07b84618fd5dc7: Status 404 returned error can't find the container with id 9fa7e03a4300dc059db31355a8eb206204e0c9a9abe056220b07b84618fd5dc7 Oct 30 13:18:36.732885 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:36.732856 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881e283c_6912_4130_8bef_7fa3f3ab9a8f.slice/crio-a3238bfa9f5bfc5f7ceb24886b65a124175fc081a21d3d765d310b151c7e2954 WatchSource:0}: Error finding container a3238bfa9f5bfc5f7ceb24886b65a124175fc081a21d3d765d310b151c7e2954: Status 404 returned error can't find the container with id a3238bfa9f5bfc5f7ceb24886b65a124175fc081a21d3d765d310b151c7e2954 Oct 30 13:18:36.739528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.739492 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.739655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.739577 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.739729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.739660 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.740331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.740298 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.740399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.740370 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.740456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.740409 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.984151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.984111 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" event={"ID":"881e283c-6912-4130-8bef-7fa3f3ab9a8f","Type":"ContainerStarted","Data":"c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f"} Oct 30 13:18:36.984663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.984181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" event={"ID":"881e283c-6912-4130-8bef-7fa3f3ab9a8f","Type":"ContainerStarted","Data":"a3238bfa9f5bfc5f7ceb24886b65a124175fc081a21d3d765d310b151c7e2954"} Oct 30 13:18:36.986202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.986172 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" event={"ID":"1516a792-85d5-46b7-b2e3-230efeb5fe70","Type":"ContainerStarted","Data":"0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474"} Oct 30 13:18:36.986304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.986226 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" event={"ID":"1516a792-85d5-46b7-b2e3-230efeb5fe70","Type":"ContainerStarted","Data":"9fa7e03a4300dc059db31355a8eb206204e0c9a9abe056220b07b84618fd5dc7"} Oct 30 13:18:36.988084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.988052 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" event={"ID":"4f0e0cae-bd47-4c25-beaa-d66a9378541e","Type":"ContainerStarted","Data":"b7164abdacdfe32609908bb0b01645091bf6faa0e3723183e698850e95a84738"} Oct 30 13:18:36.988193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.988122 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" event={"ID":"4f0e0cae-bd47-4c25-beaa-d66a9378541e","Type":"ContainerStarted","Data":"0ca1e71c00fa27b4cc60e29f150ccdb1310c8652c91ce7a993a00962bf317ddb"} Oct 30 13:18:36.988193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:36.988154 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:37.032199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.032142 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podStartSLOduration=2.032088703 podStartE2EDuration="2.032088703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:37.024753226 +0000 UTC m=+892.891371400" watchObservedRunningTime="2026-10-30 13:18:37.032088703 +0000 UTC m=+892.898706875" Oct 30 13:18:37.066062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.065995 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" podStartSLOduration=2.065977092 podStartE2EDuration="2.065977092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:37.061884326 +0000 UTC m=+892.928502501" watchObservedRunningTime="2026-10-30 13:18:37.065977092 +0000 UTC m=+892.932595266" Oct 30 13:18:37.104385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.104322 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podStartSLOduration=2.104302973 podStartE2EDuration="2.104302973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:37.103684209 +0000 UTC m=+892.970302385" watchObservedRunningTime="2026-10-30 13:18:37.104302973 +0000 UTC m=+892.970921148" Oct 30 13:18:37.123106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.123071 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:37.124262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.124235 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:37.124379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.124292 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:37.153248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.153217 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:37.154431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.154397 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:37.154572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.154473 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:37.225622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:37.214445 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:38.123328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.123291 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:38.123761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.123353 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:38.161124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.161091 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:38.161510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.161470 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:38.769647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.764959 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd"] Oct 30 13:18:38.809628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.807648 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd"] Oct 30 13:18:38.809628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.807789 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.837630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.834338 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.940641 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.940747 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.940780 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.940850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.940999 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.941045 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkgj\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.941128 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:38.955350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:38.941204 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.076403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.076311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.135751 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.135809 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.135865 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.136008 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.136060 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkgj\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.136141 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.136218 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.136624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.136317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.137459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.081045 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.137701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.137677 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:39.137835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.137812 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:39.218660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.138679 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.218903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.146138 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.219065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.181127 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.219150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.181231 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:39.219279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.219257 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:39.219361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.212311 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.219447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.209017 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.303624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.302745 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.306414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.304075 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.306414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.304627 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkgj\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.354863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.354524 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:39.744750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.732689 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:39.908623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.906699 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:39.908623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:39.907874 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:40.343669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.220713 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:40.343669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.220775 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:40.343669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.308728 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:40.343669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.308792 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:40.579930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.385098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:40.735785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.735716 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:40.942623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.879144 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:40.942623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.907672 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:40.942623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.928466 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:40.942623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.929194 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 13:18:40.960624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.947129 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:18:40.960624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.947224 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:18:40.960624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.947286 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:18:41.017190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:40.987950 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:41.017908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.017856 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:41.025632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.018156 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:41.300080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.300042 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:41.332359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.332326 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:41.332620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.332564 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:41.677689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.300455 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:41.747666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.740682 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:41.784825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.779147 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd"] Oct 30 13:18:41.905751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.887989 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:41.918422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.918389 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:42.017396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.017354 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" event={"ID":"18a1b04f-20d2-4b26-961f-05bf8646844b","Type":"ContainerStarted","Data":"a09829f43bbaaf2b7e9d2b8f4e3c5e1a1739ada463ee8590778ce6fb456b7cbf"} Oct 30 13:18:42.038704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.930740 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:42.048181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.930765 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:18:42.060996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:41.930704 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:42.068650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.061750 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:18:42.127055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.127017 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:42.127318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.127288 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:42.161548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.161513 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:42.161824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.161797 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:42.676928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.652684 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:42.767654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.740959 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:42.953698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.918683 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:42.999139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:42.999081 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:43.043763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.043719 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:43.051834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.051794 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:18:43.062451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.062400 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:43.225687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.172709 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:43.225687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.197693 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:43.225687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.174213 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:43.225687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.198113 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:43.240622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.232616 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:43.635690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.623549 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:43.657318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.657263 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" event={"ID":"18a1b04f-20d2-4b26-961f-05bf8646844b","Type":"ContainerStarted","Data":"ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273"} Oct 30 13:18:43.781678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.776883 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:18:43.931820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:43.931719 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:44.051726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.045292 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:18:44.051726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.047485 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podStartSLOduration=6.047466157 podStartE2EDuration="6.047466157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:43.965755754 +0000 UTC m=+899.832373969" watchObservedRunningTime="2026-10-30 13:18:44.047466157 +0000 UTC m=+899.914084347" Oct 30 13:18:44.073915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.053949 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:44.166621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.108754 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:44.214798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.207465 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:44.214798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.207528 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:44.214798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.207626 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:44.214798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.207646 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:44.486681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.453772 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:44.486681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.453835 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:44.820678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:44.791674 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:45.125675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.110715 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:45.188631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.137637 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:45.382731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.359336 5693 scope.go:122] "RemoveContainer" containerID="a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f" Oct 30 13:18:45.384784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.384097 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:45.384784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.384174 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:45.412067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.412023 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:45.457628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.456431 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 13:18:45.491760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.491072 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:45.491760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.491151 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:45.580624 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:18:45.579694 5693 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.146s" Oct 30 13:18:45.580624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.579744 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:45.841416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.841377 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:45.842026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.841993 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:45.864783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.864731 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:45.881123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:45.870391 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:18:46.125719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.123307 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:46.274627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.260864 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:46.274627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.260946 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:46.317700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.310223 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:18:46.464715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.361263 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:46.464715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.430059 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:46.464715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.361305 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:46.464715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.430114 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:46.609627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.513650 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:46.642635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.528085 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:46.690624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.646372 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:18:46.690958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.690932 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:18:46.864619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.864057 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 13:18:46.897627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:46.870933 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:18:47.155389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.155284 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:47.155672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.155640 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:47.258647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.240999 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:47.258647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.241154 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:47.290151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.273615 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:18:47.295061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.294142 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:18:47.322494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.322462 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:18:47.390284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.390247 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:47.399846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.393664 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:47.414737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.414654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:47.501629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.492273 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:18:47.526386 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:18:47.526347 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f\": container with ID starting with a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f not found: ID does not exist" containerID="a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f" Oct 30 13:18:47.530689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.530654 5693 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f\": container with ID starting with a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f not found: ID does not exist" Oct 30 13:18:47.582364 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:18:47.582327 5693 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f\": container with ID starting with a97e74a12749e9edd59b0d3e75c49d34664fb251367b1f5ae6373e762f3e7a5f not found: ID does not exist" Oct 30 13:18:47.583001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.582975 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:47.834618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:47.828091 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:48.071907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.071862 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:18:48.124441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.124332 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:48.124669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.124414 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:48.185561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.185508 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:48.185797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.185564 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:48.364720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.362701 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:48.364720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:48.362760 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:49.123521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.123477 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 13:18:49.124041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.123550 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 13:18:49.159621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.157532 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 13:18:49.159621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.157613 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 13:18:49.369136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.369092 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:49.369403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.369180 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:49.643387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:49.643337 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:50.264197 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.264165 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:50.267079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.267056 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:50.361624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.360051 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:50.361624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.360110 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:50.374027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.373994 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:18:50.508944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.508920 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:50.509642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.509620 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:50.510899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:50.510880 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:18:51.356155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.356115 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:51.356555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.356181 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:51.376711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.376675 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx"] Oct 30 13:18:51.382517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.382495 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.401516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.401488 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx"] Oct 30 13:18:51.407944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.407918 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b7b93a3-serving-cert\"" Oct 30 13:18:51.419411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.419379 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.419580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.419477 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.419580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.419549 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szwm\" (UniqueName: \"kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.419735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.419687 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.511523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.511482 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:51.520649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.520625 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.520791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.520681 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szwm\" (UniqueName: \"kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.520791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.520745 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.520909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.520894 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.521315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.521292 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.523286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.523269 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.523382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.523352 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.540171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.540140 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szwm\" (UniqueName: \"kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:51.668292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.668204 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:51.707940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:51.707909 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:52.102755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:52.102734 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx"] Oct 30 13:18:52.105542 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:18:52.105511 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5c7405_2694_4712_b8af_c27cf2377610.slice/crio-d330b6d35ca7099af521b2d7c90351715198924110b9fe2289a00c4fde1508de WatchSource:0}: Error finding container d330b6d35ca7099af521b2d7c90351715198924110b9fe2289a00c4fde1508de: Status 404 returned error can't find the container with id d330b6d35ca7099af521b2d7c90351715198924110b9fe2289a00c4fde1508de Oct 30 13:18:52.107056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:52.107032 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:18:52.125331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:52.125297 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" event={"ID":"2d5c7405-2694-4712-b8af-c27cf2377610","Type":"ContainerStarted","Data":"d330b6d35ca7099af521b2d7c90351715198924110b9fe2289a00c4fde1508de"} Oct 30 13:18:52.372675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:52.370657 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:18:52.372675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:52.370747 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 13:18:52.683493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:52.652736 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:53.174125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.174088 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" event={"ID":"2d5c7405-2694-4712-b8af-c27cf2377610","Type":"ContainerStarted","Data":"8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66"} Oct 30 13:18:53.175194 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.175164 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:18:53.189893 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:18:53.189853 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:18:53.217354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.217302 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" podStartSLOduration=1.777680364 podStartE2EDuration="2.217288808s" totalImagesPullingTime="439.608444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:53.214318177 +0000 UTC m=+909.080936350" watchObservedRunningTime="2026-10-30 13:18:53.217288808 +0000 UTC m=+909.083906991" Oct 30 13:18:53.377633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.376051 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:53.377633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.377613 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:53.398619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.387977 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:18:53.543041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:53.542995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:54.166864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:54.166833 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:55.313750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:55.313674 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:57.760710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:57.760664 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:18:57.789706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:57.789664 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:57.935840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:57.935788 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:57.971278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:57.971241 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.082678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:58.078701 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:58.082678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:58.079096 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:18:58.108624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:58.105689 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:59.248619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:18:59.245770 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:19:00.055743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:00.055694 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:19:00.082630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:00.073706 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:19:00.623497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:00.623457 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:02.342621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:02.337142 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:03.931330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:03.931246 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:05.617990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:05.617867 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:07.302704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:07.302651 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:08.895682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:08.895646 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:09.658878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:09.658844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:10.947220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:10.947154 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:11.184070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:11.184034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:11.629746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:11.629713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:11.987178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:11.987143 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2"] Oct 30 13:19:11.994371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:11.994347 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.023592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.023561 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ed395-openshift-default-dockercfg-7lnjc\"" Oct 30 13:19:12.029169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.029140 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2"] Oct 30 13:19:12.085775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.085740 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.139629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.129568 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.140021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.139980 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w97jn\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.164221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.164169 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.164709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.164681 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.164929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.164894 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.165187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.165158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.165366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.165334 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.165524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.165507 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.165741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.165720 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.244093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.243998 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.268968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.268931 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.269347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.269274 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.282868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.282841 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w97jn\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283075 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283204 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283454 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283626 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283883 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283881 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.283994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.283970 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.284098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.284017 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.286938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.284478 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.286938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.285309 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.286938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.285523 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.298776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.298742 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.306649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.306480 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.319145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.318612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w97jn\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.319145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.318977 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token\") pod \"e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.600241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.600155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.612305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.612277 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:12.949830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.946061 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.989634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:12.986186 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2"] Oct 30 13:19:13.015980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:13.003503 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:13.015980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:13.007686 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:13.015980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:13.007808 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:13.964504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:13.964468 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" event={"ID":"2c2f529b-d023-440d-8fd1-42351821cde3","Type":"ContainerStarted","Data":"93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f"} Oct 30 13:19:13.964708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:13.964522 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" event={"ID":"2c2f529b-d023-440d-8fd1-42351821cde3","Type":"ContainerStarted","Data":"94a1b79c7746c1577522252f9db3efb12802c052d27ff0b6d80bf4fd952764a0"} Oct 30 13:19:14.612703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:14.612669 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:14.618220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:14.618196 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:14.661208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:14.661147 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podStartSLOduration=3.661129544 podStartE2EDuration="3.661129544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:14.04400891 +0000 UTC m=+929.910627085" watchObservedRunningTime="2026-10-30 13:19:14.661129544 +0000 UTC m=+930.527747720" Oct 30 13:19:14.843621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:14.840129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:14.983620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:14.981939 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:14.983820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:14.983706 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:19:16.846900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:16.841127 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.017113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:17.017074 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.461624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:17.448191 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.515701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.515662 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.573279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.573244 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p"] Oct 30 13:19:18.578038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.578003 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:18.585610 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.585577 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j"] Oct 30 13:19:18.618183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.606788 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:18.618434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.618408 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ed395\"" Oct 30 13:19:18.661455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.660389 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p"] Oct 30 13:19:18.661455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.660664 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ed395-dockercfg-wmkbf\"" Oct 30 13:19:18.664577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.664422 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.667469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.666467 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j"] Oct 30 13:19:18.682033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.681832 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswl7\" (UniqueName: \"kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7\") pod \"payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:18.682440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.682416 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78hb\" (UniqueName: \"kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb\") pod \"payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:18.682789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.682762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:18.683562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:18.683469 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:19.145111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.143356 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:19.145111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.143523 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswl7\" (UniqueName: \"kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7\") pod \"payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:19.145111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.143731 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78hb\" (UniqueName: \"kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb\") pod \"payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:19.145111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.143773 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:19.145111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.144718 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:19.145534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.145447 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:19.182395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.182341 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78hb\" (UniqueName: \"kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb\") pod \"payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:19.182676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.182584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswl7\" (UniqueName: \"kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7\") pod \"payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:19.268898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.268863 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:19.367192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.366927 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:19.546133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.546099 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:19.746485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.746431 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p"] Oct 30 13:19:19.964385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:19.964357 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j"] Oct 30 13:19:19.965310 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:19:19.965283 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cf8abe_c39a_46f0_85db_c5870a845fe4.slice/crio-728b1394e47d461171ead419f89e09cf3ac430b8226f77b4f39e8d86858c4498 WatchSource:0}: Error finding container 728b1394e47d461171ead419f89e09cf3ac430b8226f77b4f39e8d86858c4498: Status 404 returned error can't find the container with id 728b1394e47d461171ead419f89e09cf3ac430b8226f77b4f39e8d86858c4498 Oct 30 13:19:20.164036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.163998 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" event={"ID":"e5cf8abe-c39a-46f0-85db-c5870a845fe4","Type":"ContainerStarted","Data":"672265d62c0ef83399cad8c419499bd95c4ff3657df99adfc7a8462cff5f668a"} Oct 30 13:19:20.164234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.164068 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:20.164234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.164092 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" event={"ID":"e5cf8abe-c39a-46f0-85db-c5870a845fe4","Type":"ContainerStarted","Data":"728b1394e47d461171ead419f89e09cf3ac430b8226f77b4f39e8d86858c4498"} Oct 30 13:19:20.165651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.165588 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" event={"ID":"d84fbb12-c41d-43ad-a58a-f97a41c00c73","Type":"ContainerStarted","Data":"03016b95a5fddc669fde9637678e0b5d983e13ff92be7854bd0fc37fa8b33e00"} Oct 30 13:19:20.165823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.165672 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:20.165823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.165692 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" event={"ID":"d84fbb12-c41d-43ad-a58a-f97a41c00c73","Type":"ContainerStarted","Data":"7a3b82106d5fa1382b643e14ffdc5bd0afcf9b4485c7cb89d986c9dede2748dc"} Oct 30 13:19:20.203822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.203769 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" podStartSLOduration=2.203753259 podStartE2EDuration="2.203753259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:20.20085084 +0000 UTC m=+936.067469012" watchObservedRunningTime="2026-10-30 13:19:20.203753259 +0000 UTC m=+936.070371459" Oct 30 13:19:20.259670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:20.259321 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" podStartSLOduration=2.259304275 podStartE2EDuration="2.259304275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:20.242849713 +0000 UTC m=+936.109467887" watchObservedRunningTime="2026-10-30 13:19:20.259304275 +0000 UTC m=+936.125922449" Oct 30 13:19:21.347621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:21.344898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:23.002413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:23.002377 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:23.135909 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:19:23.135859 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:19:24.692638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:24.692579 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:26.222248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.222200 5693 generic.go:356] "Generic (PLEG): container finished" podID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerID="6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62" exitCode=0 Oct 30 13:19:26.222814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.222337 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" event={"ID":"61857bbf-18b2-464f-a4f5-2d949422a51d","Type":"ContainerDied","Data":"6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62"} Oct 30 13:19:26.222915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.222883 5693 scope.go:122] "RemoveContainer" containerID="6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62" Oct 30 13:19:26.224702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.224667 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:26.224774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.224740 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:26.224867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.224777 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:26.309278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:26.309233 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:27.259335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:27.259293 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" event={"ID":"61857bbf-18b2-464f-a4f5-2d949422a51d","Type":"ContainerStarted","Data":"aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414"} Oct 30 13:19:28.024066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:28.024035 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:19:28.024868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:28.024840 5693 patch_prober.go:29] interesting pod/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:19:28.025011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:28.024902 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 13:19:28.100750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:28.100648 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:28.796842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:28.796808 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:19:29.023609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:29.023572 5693 patch_prober.go:29] interesting pod/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:19:29.023774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:29.023661 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 13:19:29.456494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:29.456452 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:29.749706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:29.749670 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:30.023799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:30.023720 5693 patch_prober.go:29] interesting pod/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:19:30.023799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:30.023771 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 13:19:31.024291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:31.024256 5693 patch_prober.go:29] interesting pod/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:19:31.024710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:31.024317 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 13:19:31.175046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:31.175018 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:19:31.175259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:31.175103 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:19:31.354803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:31.354714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:31.496648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:31.496612 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:32.024487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:32.024436 5693 patch_prober.go:29] interesting pod/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:19:32.025025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:32.024527 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 13:19:33.028262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:33.028232 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:19:33.029288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:33.029263 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:19:33.353120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:33.352944 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:35.080791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:35.080751 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:36.741937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:36.741900 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:38.712921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:38.712884 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:40.368989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:40.368951 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:41.918661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:41.918567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:43.569235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:43.569195 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:45.177296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:45.177261 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:46.850900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:46.850845 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:48.814623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:48.808908 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:50.509873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:50.509822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:52.105124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:52.105086 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:52.717468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:52.717423 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:53.058829 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:19:53.058791 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:19:53.866490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:53.866459 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:55.655163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:55.655125 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:57.857500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:57.857464 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:59.739836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:19:59.739792 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:01.414619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:01.407954 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.118766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:03.118727 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.764733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:03.764696 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz"] Oct 30 13:20:03.791592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:03.791509 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2"] Oct 30 13:20:03.929878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:03.929842 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.136393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:04.136274 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:05.016246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:05.016208 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:06.709520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:06.709481 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:08.391958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.391927 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:08.677456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.677377 5693 generic.go:356] "Generic (PLEG): container finished" podID="2c2f529b-d023-440d-8fd1-42351821cde3" containerID="93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f" exitCode=0 Oct 30 13:20:08.677456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.677451 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" event={"ID":"2c2f529b-d023-440d-8fd1-42351821cde3","Type":"ContainerDied","Data":"93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f"} Oct 30 13:20:08.677938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.677920 5693 scope.go:122] "RemoveContainer" containerID="93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f" Oct 30 13:20:08.679383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.679347 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:08.679449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.679425 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:08.679488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.679459 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:08.765764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.765733 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" podUID="b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" containerName="payload-processing" containerID="cri-o://c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0" gracePeriod=25 Oct 30 13:20:08.791843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:08.791809 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" podUID="6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" containerName="payload-pre-processing" containerID="cri-o://5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e" gracePeriod=25 Oct 30 13:20:09.107723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.107693 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:20:09.137355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.137324 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume\") pod \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " Oct 30 13:20:09.137553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.137541 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr\" (UniqueName: \"kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr\") pod \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\" (UID: \"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a\") " Oct 30 13:20:09.137794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.137767 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume" pod "b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" (UID: "b39bbefe-f2dd-4f9c-8231-10e843fa5f3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:09.138170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.138147 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.140770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.140744 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr" pod "b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" (UID: "b39bbefe-f2dd-4f9c-8231-10e843fa5f3a"). InnerVolumeSpecName "kube-api-access-kfjdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:09.158238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.158208 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:20:09.240472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.239128 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume\") pod \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " Oct 30 13:20:09.240472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.239277 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj\" (UniqueName: \"kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj\") pod \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\" (UID: \"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a\") " Oct 30 13:20:09.240472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.240056 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjdr\" (UniqueName: \"kubernetes.io/projected/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a-kube-api-access-kfjdr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.241261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.241215 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume" pod "6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" (UID: "6b77ff48-6dde-4641-ac4d-62d7f46f4f8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:09.248640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.248574 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj" pod "6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" (UID: "6b77ff48-6dde-4641-ac4d-62d7f46f4f8a"). InnerVolumeSpecName "kube-api-access-w4npj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:09.352087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.352046 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.352087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.352089 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4npj\" (UniqueName: \"kubernetes.io/projected/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a-kube-api-access-w4npj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.685994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.685908 5693 generic.go:356] "Generic (PLEG): container finished" podID="6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" containerID="5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e" exitCode=0 Oct 30 13:20:09.685994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.685952 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" Oct 30 13:20:09.686633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.685996 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" event={"ID":"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a","Type":"ContainerDied","Data":"5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e"} Oct 30 13:20:09.686633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.686044 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2" event={"ID":"6b77ff48-6dde-4641-ac4d-62d7f46f4f8a","Type":"ContainerDied","Data":"9afa15b6ed12b8892726921ca1695ae3c02e25f326e1a258bf37e6997d11afdb"} Oct 30 13:20:09.686633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.686071 5693 scope.go:122] "RemoveContainer" containerID="5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e" Oct 30 13:20:09.688311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.688282 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" event={"ID":"2c2f529b-d023-440d-8fd1-42351821cde3","Type":"ContainerStarted","Data":"246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76"} Oct 30 13:20:09.689720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.689698 5693 generic.go:356] "Generic (PLEG): container finished" podID="b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" containerID="c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0" exitCode=0 Oct 30 13:20:09.689812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.689737 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" Oct 30 13:20:09.689812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.689782 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" event={"ID":"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a","Type":"ContainerDied","Data":"c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0"} Oct 30 13:20:09.689913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.689829 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz" event={"ID":"b39bbefe-f2dd-4f9c-8231-10e843fa5f3a","Type":"ContainerDied","Data":"44e7ce44392f14260b3c0a4bbc2ee3202bffcadd0beb9b3b6e948a2a43a3face"} Oct 30 13:20:09.698702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.698684 5693 scope.go:122] "RemoveContainer" containerID="5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e" Oct 30 13:20:09.698975 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:20:09.698955 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e\": container with ID starting with 5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e not found: ID does not exist" containerID="5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e" Oct 30 13:20:09.699042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.698981 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e"} err="failed to get container status \"5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e\": rpc error: code = NotFound desc = could not find container \"5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e\": container with ID starting with 5ba4b3f0a7cc659355e5ffe187f4327906d8796f52a0dd24f276b31bc0ea5c4e not found: ID does not exist" Oct 30 13:20:09.699042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.698998 5693 scope.go:122] "RemoveContainer" containerID="c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0" Oct 30 13:20:09.718479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.718459 5693 scope.go:122] "RemoveContainer" containerID="c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0" Oct 30 13:20:09.718780 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:20:09.718763 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0\": container with ID starting with c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0 not found: ID does not exist" containerID="c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0" Oct 30 13:20:09.718843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.718789 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0"} err="failed to get container status \"c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0\": rpc error: code = NotFound desc = could not find container \"c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0\": container with ID starting with c12ccdf7d1dfff2b661de1a42bbd42b884e2c4c11a23ab09724ea82d10ac20b0 not found: ID does not exist" Oct 30 13:20:09.801427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.801388 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2"] Oct 30 13:20:09.804873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.804849 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fdad5-7d4f79c79-jsxn2"] Oct 30 13:20:09.857736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.857675 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz"] Oct 30 13:20:09.860792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:09.860758 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fdad5-6fc6758776-d9rvz"] Oct 30 13:20:10.148910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:10.143559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:10.319201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:10.319168 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" path="/var/lib/kubelet/pods/6b77ff48-6dde-4641-ac4d-62d7f46f4f8a/volumes" Oct 30 13:20:10.319705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:10.319686 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" path="/var/lib/kubelet/pods/b39bbefe-f2dd-4f9c-8231-10e843fa5f3a/volumes" Oct 30 13:20:10.613342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:10.613305 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:20:10.614243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:10.614212 5693 patch_prober.go:29] interesting pod/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 13:20:10.614398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:10.614280 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 13:20:11.613401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:11.613360 5693 patch_prober.go:29] interesting pod/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 13:20:11.613843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:11.613434 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 13:20:11.836887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:11.830074 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:12.613683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:12.613649 5693 patch_prober.go:29] interesting pod/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 13:20:12.614111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:12.613713 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 13:20:13.612879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:13.612841 5693 patch_prober.go:29] interesting pod/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 13:20:13.613064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:13.612931 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 13:20:13.703622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:13.700561 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:14.613290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:14.613256 5693 patch_prober.go:29] interesting pod/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 13:20:14.613474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:14.613306 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 13:20:14.981812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:14.981781 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:20:15.389329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:15.389188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:15.617203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:15.617172 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:20:15.800526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:15.800494 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:20:16.630409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:16.630375 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:17.146808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:17.146748 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:17.912794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:17.912752 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:18.410899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.410867 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm"] Oct 30 13:20:18.411925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.411896 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" containerName="payload-pre-processing" Oct 30 13:20:18.411925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.411918 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" containerName="payload-pre-processing" Oct 30 13:20:18.412135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.411977 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" containerName="payload-processing" Oct 30 13:20:18.412135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.411983 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" containerName="payload-processing" Oct 30 13:20:18.412135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.412121 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39bbefe-f2dd-4f9c-8231-10e843fa5f3a" containerName="payload-processing" Oct 30 13:20:18.412135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.412136 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b77ff48-6dde-4641-ac4d-62d7f46f4f8a" containerName="payload-pre-processing" Oct 30 13:20:18.417720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.417694 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.445121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.445094 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3c2202-openshift-default-dockercfg-5fk26\"" Oct 30 13:20:18.456049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.455106 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm"] Oct 30 13:20:18.510468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.510430 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:18.563493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.563448 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8772\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.563708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.563520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.563791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.563697 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.563848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.563825 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.563957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.563941 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.564017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.563975 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.564063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.564013 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.564203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.564182 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.564259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.564244 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.659622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.656968 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:18.759291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.758979 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.759291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.759103 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.759573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.759391 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.759573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.759451 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.759573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.759504 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.763620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.759738 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.763620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.760039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.763620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.760108 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8772\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.763620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.760200 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.802623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.784613 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.802623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.785023 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.802623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.786310 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.802623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.786488 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.802623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.787509 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.882582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.873228 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.912027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.911990 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8772\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.965535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.963276 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:18.987703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:18.984965 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy\") pod \"e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:19.000887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.000854 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:19.036059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.035960 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:19.494542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.494511 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm"] Oct 30 13:20:19.496627 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:19.496585 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ec8754_0b60_4957_9b3f_08f42a4e55bb.slice/crio-7824187d4eac1cd61943bd8cbdaaab17e921a697dbd178f9212f719f710fb3ea WatchSource:0}: Error finding container 7824187d4eac1cd61943bd8cbdaaab17e921a697dbd178f9212f719f710fb3ea: Status 404 returned error can't find the container with id 7824187d4eac1cd61943bd8cbdaaab17e921a697dbd178f9212f719f710fb3ea Oct 30 13:20:19.499224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.499188 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:19.499319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.499260 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:19.499375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.499315 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:19.985096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.985067 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" event={"ID":"e3ec8754-0b60-4957-9b3f-08f42a4e55bb","Type":"ContainerStarted","Data":"2aacde4f17ec3cf65a8fdba6b3e6d7dafaaf47b23ea259f9584356337292f50b"} Oct 30 13:20:19.985583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:19.985127 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" event={"ID":"e3ec8754-0b60-4957-9b3f-08f42a4e55bb","Type":"ContainerStarted","Data":"7824187d4eac1cd61943bd8cbdaaab17e921a697dbd178f9212f719f710fb3ea"} Oct 30 13:20:20.036473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:20.036440 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:20.040992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:20.040940 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podStartSLOduration=2.040925381 podStartE2EDuration="2.040925381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:20.036897963 +0000 UTC m=+995.903516137" watchObservedRunningTime="2026-10-30 13:20:20.040925381 +0000 UTC m=+995.907543553" Oct 30 13:20:20.044774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:20.044744 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:20.690778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:20.690743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:20.992779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:20.992753 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:20.993729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:20.993711 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:20:22.456985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:22.456955 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:22.611541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:22.611504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:23.058943 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:20:23.058905 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:20:23.516055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:23.516026 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:24.155646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.155612 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:24.226255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.226225 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w"] Oct 30 13:20:24.233971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.233934 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.257446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.257416 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3c2202\"" Oct 30 13:20:24.262746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.262717 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w"] Oct 30 13:20:24.276916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.276888 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3c2202-dockercfg-qj92p\"" Oct 30 13:20:24.312172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.312138 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:24.320059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.320032 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4"] Oct 30 13:20:24.324433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.324413 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.373630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.368742 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx55n\" (UniqueName: \"kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n\") pod \"payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.373630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.368974 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.396268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.396222 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4"] Oct 30 13:20:24.515621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.491869 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.542680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.541731 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx55n\" (UniqueName: \"kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n\") pod \"payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.542680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.541852 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbrx\" (UniqueName: \"kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx\") pod \"payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.542680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.542034 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.543288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.543213 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.600375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.600336 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx55n\" (UniqueName: \"kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n\") pod \"payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.620221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.620163 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:24.688624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.649469 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.689339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.689307 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbrx\" (UniqueName: \"kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx\") pod \"payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.705636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.667912 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.748322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.748284 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbrx\" (UniqueName: \"kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx\") pod \"payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:24.861408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.861290 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:24.971743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:24.971670 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:25.173972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:25.173946 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w"] Oct 30 13:20:25.175628 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:25.175583 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85675983_0f0d_4528_958b_111d23691544.slice/crio-dfa9609fec68fc13b374b98d01487b5431740bfad9d46145c29fa00f082ea171 WatchSource:0}: Error finding container dfa9609fec68fc13b374b98d01487b5431740bfad9d46145c29fa00f082ea171: Status 404 returned error can't find the container with id dfa9609fec68fc13b374b98d01487b5431740bfad9d46145c29fa00f082ea171 Oct 30 13:20:25.287010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:25.286981 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4"] Oct 30 13:20:25.289918 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:25.289893 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7127f9b_61cc_45c8_8fdd_4aecc651e307.slice/crio-8939dfa5a4140b3d0c4b1f61b514cf3d1d5cb904cd448b8886edf17767373d7f WatchSource:0}: Error finding container 8939dfa5a4140b3d0c4b1f61b514cf3d1d5cb904cd448b8886edf17767373d7f: Status 404 returned error can't find the container with id 8939dfa5a4140b3d0c4b1f61b514cf3d1d5cb904cd448b8886edf17767373d7f Oct 30 13:20:26.030388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.030348 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" event={"ID":"d7127f9b-61cc-45c8-8fdd-4aecc651e307","Type":"ContainerStarted","Data":"5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921"} Oct 30 13:20:26.030889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.030411 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" event={"ID":"d7127f9b-61cc-45c8-8fdd-4aecc651e307","Type":"ContainerStarted","Data":"8939dfa5a4140b3d0c4b1f61b514cf3d1d5cb904cd448b8886edf17767373d7f"} Oct 30 13:20:26.030889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.030439 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:26.032125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.032099 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" event={"ID":"85675983-0f0d-4528-958b-111d23691544","Type":"ContainerStarted","Data":"757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30"} Oct 30 13:20:26.032246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.032151 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" event={"ID":"85675983-0f0d-4528-958b-111d23691544","Type":"ContainerStarted","Data":"dfa9609fec68fc13b374b98d01487b5431740bfad9d46145c29fa00f082ea171"} Oct 30 13:20:26.032246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.032180 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:26.071015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.070961 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" podStartSLOduration=2.070947223 podStartE2EDuration="2.070947223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:26.069120331 +0000 UTC m=+1001.935738503" watchObservedRunningTime="2026-10-30 13:20:26.070947223 +0000 UTC m=+1001.937565395" Oct 30 13:20:26.106734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.106619 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" podStartSLOduration=2.106587338 podStartE2EDuration="2.106587338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:26.105583293 +0000 UTC m=+1001.972201467" watchObservedRunningTime="2026-10-30 13:20:26.106587338 +0000 UTC m=+1001.973205512" Oct 30 13:20:26.281491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:26.281417 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:27.150285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:27.150245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:27.298792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:27.298755 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:27.435587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:27.435504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:27.762671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:27.762639 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:29.147622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:29.144675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:30.865584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:30.858270 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.012305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:32.012273 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.456906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:32.456824 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.679623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:32.679067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:34.102827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:34.102791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:34.526511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:34.526474 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:34.678857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:34.678814 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:34.803096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:34.802996 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n"] Oct 30 13:20:34.803351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:34.803319 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" containerID="cri-o://aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414" gracePeriod=30 Oct 30 13:20:34.872310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:34.872273 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:35.620117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:35.620084 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:35.764482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:35.764441 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:36.079076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:36.078961 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:36.236863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:36.236830 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm"] Oct 30 13:20:36.407640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:36.407510 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:36.553659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:36.553621 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:37.057140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:37.055867 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:20:37.057861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:37.057840 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:20:38.058826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.058794 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:38.148097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.147656 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2"] Oct 30 13:20:38.232731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.232696 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2"] Oct 30 13:20:38.233072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.233053 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.341674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.294051 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8ac28e\"" Oct 30 13:20:38.341674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.294372 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8ac28e-dockercfg-zgqlg\"" Oct 30 13:20:38.341674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.303359 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.341674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.303465 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twr7n\" (UniqueName: \"kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.423198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.421642 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:38.423198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.423000 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.423198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.423131 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twr7n\" (UniqueName: \"kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.425928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.425216 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.768636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.756312 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twr7n\" (UniqueName: \"kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:38.768636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.763623 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:20:38.824772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.824733 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:39.021688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:38.932890 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65"] Oct 30 13:20:39.052837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.041327 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65"] Oct 30 13:20:39.052837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.041456 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.233735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.227865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.233735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.227945 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gsl\" (UniqueName: \"kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl\") pod \"payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.405657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.403441 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.405657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.403514 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gsl\" (UniqueName: \"kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl\") pod \"payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.438628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.414426 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.438628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.436419 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.498350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.498313 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gsl\" (UniqueName: \"kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl\") pod \"payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:39.559681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.556494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.685180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:39.684885 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:40.321998 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:40.318543 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8bd06d_82de_46bc_9592_21699f8fb529.slice/crio-52ce1276d7287f3dbbff7ab29e355d3624d810e6010330bcc658af27663db073 WatchSource:0}: Error finding container 52ce1276d7287f3dbbff7ab29e355d3624d810e6010330bcc658af27663db073: Status 404 returned error can't find the container with id 52ce1276d7287f3dbbff7ab29e355d3624d810e6010330bcc658af27663db073 Oct 30 13:20:40.331654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.331295 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65"] Oct 30 13:20:40.464622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.458460 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2"] Oct 30 13:20:40.472218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.472192 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:20:40.475290 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:40.475039 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d79bd41_09af_4f3a_be24_b1512652335d.slice/crio-ad0e3f53745dc9fcb828fb2d04ba5c580f70297541f32f78442171066f82fdb8 WatchSource:0}: Error finding container ad0e3f53745dc9fcb828fb2d04ba5c580f70297541f32f78442171066f82fdb8: Status 404 returned error can't find the container with id ad0e3f53745dc9fcb828fb2d04ba5c580f70297541f32f78442171066f82fdb8 Oct 30 13:20:40.561864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.561829 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.561895 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562030 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562103 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562341 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562375 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:40.562500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562452 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:40.562565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562525 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562643 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:40.562676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562652 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562752 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy\") pod \"61857bbf-18b2-464f-a4f5-2d949422a51d\" (UID: \"61857bbf-18b2-464f-a4f5-2d949422a51d\") " Oct 30 13:20:40.562825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562765 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:40.562825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.562807 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:40.563449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.563429 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.563449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.563448 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.563624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.563457 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.563624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.563468 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.563624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.563481 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61857bbf-18b2-464f-a4f5-2d949422a51d-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.564740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.564712 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:40.565114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.565091 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:20:40.565204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.565114 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "kube-api-access-qdvh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:40.565204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.565196 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy" pod "61857bbf-18b2-464f-a4f5-2d949422a51d" (UID: "61857bbf-18b2-464f-a4f5-2d949422a51d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:40.682675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.666678 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvh5\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-kube-api-access-qdvh5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.682675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.666724 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.682675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.666749 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.682675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.666763 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61857bbf-18b2-464f-a4f5-2d949422a51d-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:40.970928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.970827 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" event={"ID":"2d79bd41-09af-4f3a-be24-b1512652335d","Type":"ContainerStarted","Data":"c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4"} Oct 30 13:20:40.970928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.970898 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:40.970928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.970922 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" event={"ID":"2d79bd41-09af-4f3a-be24-b1512652335d","Type":"ContainerStarted","Data":"ad0e3f53745dc9fcb828fb2d04ba5c580f70297541f32f78442171066f82fdb8"} Oct 30 13:20:40.972657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.972591 5693 generic.go:356] "Generic (PLEG): container finished" podID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerID="aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414" exitCode=0 Oct 30 13:20:40.972657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.972648 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" Oct 30 13:20:40.972859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.972628 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" event={"ID":"61857bbf-18b2-464f-a4f5-2d949422a51d","Type":"ContainerDied","Data":"aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414"} Oct 30 13:20:40.972859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.972757 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n" event={"ID":"61857bbf-18b2-464f-a4f5-2d949422a51d","Type":"ContainerDied","Data":"78114f89c5f53637514f0f0904c869c22f268e14c71825bd3735dfd19cad3da0"} Oct 30 13:20:40.972859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.972779 5693 scope.go:122] "RemoveContainer" containerID="aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414" Oct 30 13:20:40.974657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.974634 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" event={"ID":"ae8bd06d-82de-46bc-9592-21699f8fb529","Type":"ContainerStarted","Data":"9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4"} Oct 30 13:20:40.974787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.974682 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" event={"ID":"ae8bd06d-82de-46bc-9592-21699f8fb529","Type":"ContainerStarted","Data":"52ce1276d7287f3dbbff7ab29e355d3624d810e6010330bcc658af27663db073"} Oct 30 13:20:40.974787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.974711 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:40.983494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.983478 5693 scope.go:122] "RemoveContainer" containerID="6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62" Oct 30 13:20:40.992964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.992944 5693 scope.go:122] "RemoveContainer" containerID="aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414" Oct 30 13:20:40.993219 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:20:40.993199 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414\": container with ID starting with aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414 not found: ID does not exist" containerID="aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414" Oct 30 13:20:40.993265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.993227 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414"} err="failed to get container status \"aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414\": rpc error: code = NotFound desc = could not find container \"aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414\": container with ID starting with aaf6bf51640915b5e78ba43ece3d90b21069a7f87e22a4afeb8a7b65d8d26414 not found: ID does not exist" Oct 30 13:20:40.993265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.993245 5693 scope.go:122] "RemoveContainer" containerID="6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62" Oct 30 13:20:40.993477 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:20:40.993461 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62\": container with ID starting with 6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62 not found: ID does not exist" containerID="6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62" Oct 30 13:20:40.993514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:40.993482 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62"} err="failed to get container status \"6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62\": rpc error: code = NotFound desc = could not find container \"6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62\": container with ID starting with 6366ae08e2a481ca8717e45924dd061fb55001b049ab17232462538ebd0f4f62 not found: ID does not exist" Oct 30 13:20:41.011240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:41.011172 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" podStartSLOduration=3.011149401 podStartE2EDuration="3.011149401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:41.008793574 +0000 UTC m=+1016.875411748" watchObservedRunningTime="2026-10-30 13:20:41.011149401 +0000 UTC m=+1016.877767577" Oct 30 13:20:41.049744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:41.049714 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n"] Oct 30 13:20:41.061120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:41.061087 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9fdad5-openshift-default-79fd8fd8cf-42c2n"] Oct 30 13:20:41.089138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:41.089080 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" podStartSLOduration=3.08906692 podStartE2EDuration="3.08906692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:41.088839411 +0000 UTC m=+1016.955457584" watchObservedRunningTime="2026-10-30 13:20:41.08906692 +0000 UTC m=+1016.955685133" Oct 30 13:20:41.259621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:41.258972 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerName="payload-processing" containerID="cri-o://b7164abdacdfe32609908bb0b01645091bf6faa0e3723183e698850e95a84738" gracePeriod=25 Oct 30 13:20:41.605993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:41.605904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:42.256039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:42.256006 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" path="/var/lib/kubelet/pods/61857bbf-18b2-464f-a4f5-2d949422a51d/volumes" Oct 30 13:20:43.323699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:43.315884 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:43.943057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:43.943004 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:44.096844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:44.096791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:45.016668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:45.016634 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:45.783178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:45.783144 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j"] Oct 30 13:20:45.813433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:45.813392 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p"] Oct 30 13:20:45.943791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:45.943736 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:46.092162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:46.092050 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:47.000229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:47.000197 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:48.001471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:48.001420 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:20:48.595089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:48.595056 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.099481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.099441 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2"] Oct 30 13:20:50.100505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.100482 5693 cpu_manager.go:414] "containerMap: removing container" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" Oct 30 13:20:50.100505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.100504 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" Oct 30 13:20:50.100778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.100751 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" Oct 30 13:20:50.106138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.106117 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.125363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.125328 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2"] Oct 30 13:20:50.132107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.131846 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8ac28e-serving-cert\"" Oct 30 13:20:50.174407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.174369 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.174743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.174712 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.174905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.174796 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.175004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.174978 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hlc\" (UniqueName: \"kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.232006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.231968 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.276263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.276232 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hlc\" (UniqueName: \"kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.276415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.276365 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.276415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.276408 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.276519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.276426 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.277050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.277028 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.279035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.279010 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.279136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.279053 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.296466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.296441 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hlc\" (UniqueName: \"kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc\") pod \"maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.370656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.370495 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.420672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.420376 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:50.766175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.766150 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2"] Oct 30 13:20:50.768249 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:50.768219 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1285ceff_bd1a_4d05_9528_63b3328b6002.slice/crio-f3e9c76ff16b1ced76d9dd905ccc6fbfa2373411bb845128975aa34e7088f05f WatchSource:0}: Error finding container f3e9c76ff16b1ced76d9dd905ccc6fbfa2373411bb845128975aa34e7088f05f: Status 404 returned error can't find the container with id f3e9c76ff16b1ced76d9dd905ccc6fbfa2373411bb845128975aa34e7088f05f Oct 30 13:20:50.784273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.784244 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" podUID="e5cf8abe-c39a-46f0-85db-c5870a845fe4" containerName="payload-processing" containerID="cri-o://672265d62c0ef83399cad8c419499bd95c4ff3657df99adfc7a8462cff5f668a" gracePeriod=25 Oct 30 13:20:50.814351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:50.814313 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" podUID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" containerName="payload-pre-processing" containerID="cri-o://03016b95a5fddc669fde9637678e0b5d983e13ff92be7854bd0fc37fa8b33e00" gracePeriod=25 Oct 30 13:20:51.137707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.137671 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" event={"ID":"1285ceff-bd1a-4d05-9528-63b3328b6002","Type":"ContainerStarted","Data":"f3e9c76ff16b1ced76d9dd905ccc6fbfa2373411bb845128975aa34e7088f05f"} Oct 30 13:20:51.148920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.148895 5693 generic.go:356] "Generic (PLEG): container finished" podID="e5cf8abe-c39a-46f0-85db-c5870a845fe4" containerID="672265d62c0ef83399cad8c419499bd95c4ff3657df99adfc7a8462cff5f668a" exitCode=0 Oct 30 13:20:51.149146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.148966 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" event={"ID":"e5cf8abe-c39a-46f0-85db-c5870a845fe4","Type":"ContainerDied","Data":"672265d62c0ef83399cad8c419499bd95c4ff3657df99adfc7a8462cff5f668a"} Oct 30 13:20:51.153724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.153696 5693 generic.go:356] "Generic (PLEG): container finished" podID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" containerID="03016b95a5fddc669fde9637678e0b5d983e13ff92be7854bd0fc37fa8b33e00" exitCode=0 Oct 30 13:20:51.153864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.153750 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" event={"ID":"d84fbb12-c41d-43ad-a58a-f97a41c00c73","Type":"ContainerDied","Data":"03016b95a5fddc669fde9637678e0b5d983e13ff92be7854bd0fc37fa8b33e00"} Oct 30 13:20:51.166752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.165919 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:20:51.191175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.191143 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:20:51.195729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.195706 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume\") pod \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " Oct 30 13:20:51.196012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.195998 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb\" (UniqueName: \"kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb\") pod \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\" (UID: \"e5cf8abe-c39a-46f0-85db-c5870a845fe4\") " Oct 30 13:20:51.196063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.196023 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume" pod "e5cf8abe-c39a-46f0-85db-c5870a845fe4" (UID: "e5cf8abe-c39a-46f0-85db-c5870a845fe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:51.196427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.196401 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cf8abe-c39a-46f0-85db-c5870a845fe4-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:51.198193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.198139 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb" pod "e5cf8abe-c39a-46f0-85db-c5870a845fe4" (UID: "e5cf8abe-c39a-46f0-85db-c5870a845fe4"). InnerVolumeSpecName "kube-api-access-z78hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:51.297962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.297706 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7\" (UniqueName: \"kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7\") pod \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " Oct 30 13:20:51.297962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.297913 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume\") pod \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\" (UID: \"d84fbb12-c41d-43ad-a58a-f97a41c00c73\") " Oct 30 13:20:51.298637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.298499 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z78hb\" (UniqueName: \"kubernetes.io/projected/e5cf8abe-c39a-46f0-85db-c5870a845fe4-kube-api-access-z78hb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:51.322859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.298899 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume" pod "d84fbb12-c41d-43ad-a58a-f97a41c00c73" (UID: "d84fbb12-c41d-43ad-a58a-f97a41c00c73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:51.322859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.301923 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7" pod "d84fbb12-c41d-43ad-a58a-f97a41c00c73" (UID: "d84fbb12-c41d-43ad-a58a-f97a41c00c73"). InnerVolumeSpecName "kube-api-access-fswl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:51.438047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.414172 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswl7\" (UniqueName: \"kubernetes.io/projected/d84fbb12-c41d-43ad-a58a-f97a41c00c73-kube-api-access-fswl7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:51.458623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.451159 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d84fbb12-c41d-43ad-a58a-f97a41c00c73-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:20:51.986964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.986929 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:20:51.987140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:51.986999 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:20:52.037795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.037744 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:52.199079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.199032 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" podUID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.99:50052\" within 1s: context deadline exceeded" Oct 30 13:20:52.199586 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:20:52.199394 5693 logging.go:55] [core] [Channel #1577 SubChannel #1578]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.99:50052", ServerName: "10.134.0.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.99:50052: operation was canceled" Oct 30 13:20:52.227129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.227082 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" event={"ID":"1285ceff-bd1a-4d05-9528-63b3328b6002","Type":"ContainerStarted","Data":"d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870"} Oct 30 13:20:52.228749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.228720 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:52.230421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.230400 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" Oct 30 13:20:52.231213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.230422 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j" event={"ID":"e5cf8abe-c39a-46f0-85db-c5870a845fe4","Type":"ContainerDied","Data":"728b1394e47d461171ead419f89e09cf3ac430b8226f77b4f39e8d86858c4498"} Oct 30 13:20:52.231213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.231130 5693 scope.go:122] "RemoveContainer" containerID="672265d62c0ef83399cad8c419499bd95c4ff3657df99adfc7a8462cff5f668a" Oct 30 13:20:52.239477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.239418 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" event={"ID":"d84fbb12-c41d-43ad-a58a-f97a41c00c73","Type":"ContainerDied","Data":"7a3b82106d5fa1382b643e14ffdc5bd0afcf9b4485c7cb89d986c9dede2748dc"} Oct 30 13:20:52.239577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.239517 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p" Oct 30 13:20:52.257379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.257044 5693 scope.go:122] "RemoveContainer" containerID="03016b95a5fddc669fde9637678e0b5d983e13ff92be7854bd0fc37fa8b33e00" Oct 30 13:20:52.285737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.285692 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" podStartSLOduration=1.9384343899999998 podStartE2EDuration="2.285678226s" totalImagesPullingTime="347.243836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:52.282528087 +0000 UTC m=+1028.149146262" watchObservedRunningTime="2026-10-30 13:20:52.285678226 +0000 UTC m=+1028.152296410" Oct 30 13:20:52.319213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.319192 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p"] Oct 30 13:20:52.322816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.322792 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed395-6fb87b566c-pjn4p"] Oct 30 13:20:52.356029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.356001 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j"] Oct 30 13:20:52.358996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.358976 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed395-7dc894cdb8-nnp4j"] Oct 30 13:20:52.837179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:52.837144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:53.057471 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:20:53.057431 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:20:53.677002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:53.676964 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:54.255904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:54.255867 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" path="/var/lib/kubelet/pods/d84fbb12-c41d-43ad-a58a-f97a41c00c73/volumes" Oct 30 13:20:54.256321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:54.256304 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cf8abe-c39a-46f0-85db-c5870a845fe4" path="/var/lib/kubelet/pods/e5cf8abe-c39a-46f0-85db-c5870a845fe4/volumes" Oct 30 13:20:55.305628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:55.303291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.122416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:57.122363 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.952661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:57.952591 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:58.013588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:58.013546 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:20:58.619039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:58.619004 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:58.762770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:58.762737 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:59.293482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:59.293450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:59.303616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:59.303572 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:20:59.618972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:20:59.618884 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:00.380992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.380958 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx"] Oct 30 13:21:00.381710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.381676 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" podUID="2d5c7405-2694-4712-b8af-c27cf2377610" containerName="maas-api" containerID="cri-o://8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66" gracePeriod=30 Oct 30 13:21:00.482065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.482024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:00.854481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.854454 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:21:00.962202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.962171 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls\") pod \"2d5c7405-2694-4712-b8af-c27cf2377610\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " Oct 30 13:21:00.962432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.962236 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs\") pod \"2d5c7405-2694-4712-b8af-c27cf2377610\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " Oct 30 13:21:00.962432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.962289 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm\" (UniqueName: \"kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm\") pod \"2d5c7405-2694-4712-b8af-c27cf2377610\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " Oct 30 13:21:00.962432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.962312 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca\") pod \"2d5c7405-2694-4712-b8af-c27cf2377610\" (UID: \"2d5c7405-2694-4712-b8af-c27cf2377610\") " Oct 30 13:21:00.962799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.962766 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca" pod "2d5c7405-2694-4712-b8af-c27cf2377610" (UID: "2d5c7405-2694-4712-b8af-c27cf2377610"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:00.963213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.963189 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5c7405-2694-4712-b8af-c27cf2377610-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:00.964435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.964410 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs" pod "2d5c7405-2694-4712-b8af-c27cf2377610" (UID: "2d5c7405-2694-4712-b8af-c27cf2377610"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:21:00.964525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.964463 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm" pod "2d5c7405-2694-4712-b8af-c27cf2377610" (UID: "2d5c7405-2694-4712-b8af-c27cf2377610"). InnerVolumeSpecName "kube-api-access-5szwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:00.964525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:00.964472 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls" pod "2d5c7405-2694-4712-b8af-c27cf2377610" (UID: "2d5c7405-2694-4712-b8af-c27cf2377610"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:21:01.084771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.070792 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:01.084771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.070830 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5c7405-2694-4712-b8af-c27cf2377610-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:01.084771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.070845 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5szwm\" (UniqueName: \"kubernetes.io/projected/2d5c7405-2694-4712-b8af-c27cf2377610-kube-api-access-5szwm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:01.166571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.166536 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:01.342672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.338587 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:01.425579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.424305 5693 generic.go:356] "Generic (PLEG): container finished" podID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerID="b7164abdacdfe32609908bb0b01645091bf6faa0e3723183e698850e95a84738" exitCode=0 Oct 30 13:21:01.425579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.424425 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" event={"ID":"4f0e0cae-bd47-4c25-beaa-d66a9378541e","Type":"ContainerDied","Data":"b7164abdacdfe32609908bb0b01645091bf6faa0e3723183e698850e95a84738"} Oct 30 13:21:01.428079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.428044 5693 generic.go:356] "Generic (PLEG): container finished" podID="2d5c7405-2694-4712-b8af-c27cf2377610" containerID="8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66" exitCode=0 Oct 30 13:21:01.428191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.428103 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" Oct 30 13:21:01.428191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.428119 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" event={"ID":"2d5c7405-2694-4712-b8af-c27cf2377610","Type":"ContainerDied","Data":"8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66"} Oct 30 13:21:01.428191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.428174 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx" event={"ID":"2d5c7405-2694-4712-b8af-c27cf2377610","Type":"ContainerDied","Data":"d330b6d35ca7099af521b2d7c90351715198924110b9fe2289a00c4fde1508de"} Oct 30 13:21:01.428447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.428203 5693 scope.go:122] "RemoveContainer" containerID="8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66" Oct 30 13:21:01.443225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.443139 5693 scope.go:122] "RemoveContainer" containerID="8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66" Oct 30 13:21:01.443743 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:01.443720 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66\": container with ID starting with 8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66 not found: ID does not exist" containerID="8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66" Oct 30 13:21:01.443848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.443750 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66"} err="failed to get container status \"8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66\": rpc error: code = NotFound desc = could not find container \"8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66\": container with ID starting with 8b96f5f9f4cc75a2f8d422cb9204a777eb37431e9ee9c31633f191b866076f66 not found: ID does not exist" Oct 30 13:21:01.492333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.492263 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx"] Oct 30 13:21:01.504853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.504818 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-cdzbx"] Oct 30 13:21:01.507052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.507018 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:01.747626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.734827 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:21:01.854549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.854499 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4\" (UniqueName: \"kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4\") pod \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " Oct 30 13:21:01.854824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.854562 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume\") pod \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\" (UID: \"4f0e0cae-bd47-4c25-beaa-d66a9378541e\") " Oct 30 13:21:01.854959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.854936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume" pod "4f0e0cae-bd47-4c25-beaa-d66a9378541e" (UID: "4f0e0cae-bd47-4c25-beaa-d66a9378541e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:01.855321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.855298 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0e0cae-bd47-4c25-beaa-d66a9378541e-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:01.856961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.856932 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4" pod "4f0e0cae-bd47-4c25-beaa-d66a9378541e" (UID: "4f0e0cae-bd47-4c25-beaa-d66a9378541e"). InnerVolumeSpecName "kube-api-access-tdxj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:01.957313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:01.957218 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxj4\" (UniqueName: \"kubernetes.io/projected/4f0e0cae-bd47-4c25-beaa-d66a9378541e-kube-api-access-tdxj4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:02.262557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:02.262516 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5c7405-2694-4712-b8af-c27cf2377610" path="/var/lib/kubelet/pods/2d5c7405-2694-4712-b8af-c27cf2377610/volumes" Oct 30 13:21:02.458965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:02.458936 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" Oct 30 13:21:02.458965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:02.458942 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm" event={"ID":"4f0e0cae-bd47-4c25-beaa-d66a9378541e","Type":"ContainerDied","Data":"0ca1e71c00fa27b4cc60e29f150ccdb1310c8652c91ce7a993a00962bf317ddb"} Oct 30 13:21:02.459507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:02.458998 5693 scope.go:122] "RemoveContainer" containerID="b7164abdacdfe32609908bb0b01645091bf6faa0e3723183e698850e95a84738" Oct 30 13:21:02.557916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:02.557827 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm"] Oct 30 13:21:02.577263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:02.577226 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-gj8rm"] Oct 30 13:21:03.476266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:03.476236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:03.662530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:03.662491 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:03.882241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:03.882155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:04.263259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:04.263224 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" path="/var/lib/kubelet/pods/4f0e0cae-bd47-4c25-beaa-d66a9378541e/volumes" Oct 30 13:21:05.117688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.117645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:05.197235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.197196 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd"] Oct 30 13:21:05.198262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198245 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2d5c7405-2694-4712-b8af-c27cf2377610" containerName="maas-api" Oct 30 13:21:05.198316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198264 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5c7405-2694-4712-b8af-c27cf2377610" containerName="maas-api" Oct 30 13:21:05.198316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198288 5693 cpu_manager.go:414] "containerMap: removing container" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" Oct 30 13:21:05.198316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198296 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" Oct 30 13:21:05.198316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198310 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" containerName="payload-pre-processing" Oct 30 13:21:05.198455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198317 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" containerName="payload-pre-processing" Oct 30 13:21:05.198455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198344 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e5cf8abe-c39a-46f0-85db-c5870a845fe4" containerName="payload-processing" Oct 30 13:21:05.198455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198351 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cf8abe-c39a-46f0-85db-c5870a845fe4" containerName="payload-processing" Oct 30 13:21:05.198455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198378 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerName="payload-processing" Oct 30 13:21:05.198455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198385 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerName="payload-processing" Oct 30 13:21:05.198635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198513 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0e0cae-bd47-4c25-beaa-d66a9378541e" containerName="payload-processing" Oct 30 13:21:05.198635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198524 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="61857bbf-18b2-464f-a4f5-2d949422a51d" containerName="istio-proxy" Oct 30 13:21:05.198635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198540 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5c7405-2694-4712-b8af-c27cf2377610" containerName="maas-api" Oct 30 13:21:05.198635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198550 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84fbb12-c41d-43ad-a58a-f97a41c00c73" containerName="payload-pre-processing" Oct 30 13:21:05.198635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.198563 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cf8abe-c39a-46f0-85db-c5870a845fe4" containerName="payload-processing" Oct 30 13:21:05.204159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.204130 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.220356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.220331 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd"] Oct 30 13:21:05.257988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.257687 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7c0e28\"" Oct 30 13:21:05.262222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.262188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:05.265949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.265927 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh"] Oct 30 13:21:05.269720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.269689 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7c0e28-dockercfg-dzznf\"" Oct 30 13:21:05.270541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.270519 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.282334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.282309 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh"] Oct 30 13:21:05.349620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.348342 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.349620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.348615 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bm2\" (UniqueName: \"kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2\") pod \"payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.588689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.587947 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.588689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.588209 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.588689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.588255 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bm2\" (UniqueName: \"kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2\") pod \"payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.588689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.588361 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zb6m\" (UniqueName: \"kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m\") pod \"payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.680621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.590037 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.690920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.690882 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.691255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.691235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zb6m\" (UniqueName: \"kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m\") pod \"payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.692636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.692579 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.698372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.698346 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bm2\" (UniqueName: \"kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2\") pod \"payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.718342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.718311 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zb6m\" (UniqueName: \"kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m\") pod \"payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:05.754317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.754279 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:05.830926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.830889 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:05.952710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:05.950971 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:06.388077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.388051 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:06.388467 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:06.388260 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfd37c2_e2ce_4bac_ae6d_0c8247b3f7d4.slice/crio-1c9fc405951bcb552f1c3d7175f25c88593e1a8457b84dcc586e3963c8e8126f WatchSource:0}: Error finding container 1c9fc405951bcb552f1c3d7175f25c88593e1a8457b84dcc586e3963c8e8126f: Status 404 returned error can't find the container with id 1c9fc405951bcb552f1c3d7175f25c88593e1a8457b84dcc586e3963c8e8126f Oct 30 13:21:06.405297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.405268 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd"] Oct 30 13:21:06.405402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.405329 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh"] Oct 30 13:21:06.700089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.699995 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" event={"ID":"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4","Type":"ContainerStarted","Data":"00d4a4b6040fa64fe1bef31600100eba3706eccd3c5d76325ad63fc0077f503a"} Oct 30 13:21:06.700089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.700063 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" event={"ID":"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4","Type":"ContainerStarted","Data":"1c9fc405951bcb552f1c3d7175f25c88593e1a8457b84dcc586e3963c8e8126f"} Oct 30 13:21:06.700278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.700255 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:06.701623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.701585 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" event={"ID":"687a6f30-a6c8-49c9-ac79-8da51b65e21f","Type":"ContainerStarted","Data":"99ee0112a9b9ff0e8fc20f194ff1f0946fde1d9c5b1a34c6c85891c86d30aeee"} Oct 30 13:21:06.701740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.701638 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" event={"ID":"687a6f30-a6c8-49c9-ac79-8da51b65e21f","Type":"ContainerStarted","Data":"346d2d5fe120c000a8e481fbf6b013ac9107408307ae348a6c087d46821c7677"} Oct 30 13:21:06.701740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.701731 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:06.748622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.747863 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" podStartSLOduration=1.747843927 podStartE2EDuration="1.747843927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:06.745024129 +0000 UTC m=+1042.611642303" watchObservedRunningTime="2026-10-30 13:21:06.747843927 +0000 UTC m=+1042.614462103" Oct 30 13:21:06.785419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:06.785362 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" podStartSLOduration=1.785348882 podStartE2EDuration="1.785348882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:06.783585397 +0000 UTC m=+1042.650203570" watchObservedRunningTime="2026-10-30 13:21:06.785348882 +0000 UTC m=+1042.651967055" Oct 30 13:21:08.278582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:08.278546 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:09.820450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:09.820403 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:10.421070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:10.421029 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:10.620158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:10.620119 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:10.759375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:10.757123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:11.045110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:11.045014 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65"] Oct 30 13:21:11.075934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:11.075897 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2"] Oct 30 13:21:11.134697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:11.134651 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:11.304675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:11.296045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:11.469183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:11.469153 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:12.574574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.574525 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:12.723625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.719218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:12.829083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.828993 5693 generic.go:356] "Generic (PLEG): container finished" podID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerID="2aacde4f17ec3cf65a8fdba6b3e6d7dafaaf47b23ea259f9584356337292f50b" exitCode=0 Oct 30 13:21:12.829083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.829070 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" event={"ID":"e3ec8754-0b60-4957-9b3f-08f42a4e55bb","Type":"ContainerDied","Data":"2aacde4f17ec3cf65a8fdba6b3e6d7dafaaf47b23ea259f9584356337292f50b"} Oct 30 13:21:12.829498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.829481 5693 scope.go:122] "RemoveContainer" containerID="2aacde4f17ec3cf65a8fdba6b3e6d7dafaaf47b23ea259f9584356337292f50b" Oct 30 13:21:12.831003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.830972 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:12.831120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.831056 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:12.831120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:12.831102 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:13.314216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.314186 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:13.397733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.397700 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j"] Oct 30 13:21:13.402423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.402401 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.416967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.416921 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b7b93a3\"" Oct 30 13:21:13.419967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.419938 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j"] Oct 30 13:21:13.442981 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.442919 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c"] Oct 30 13:21:13.443631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.443587 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b7b93a3-dockercfg-bsd7f\"" Oct 30 13:21:13.448999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.448974 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:13.476503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.476468 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:13.489736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.489702 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.489904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.489829 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfmv\" (UniqueName: \"kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.489972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.489932 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:13.490029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.489974 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck29m\" (UniqueName: \"kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:13.500834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.500805 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c"] Oct 30 13:21:13.720678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.719770 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:13.720678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.719863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck29m\" (UniqueName: \"kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:13.720678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.720135 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.720678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.720245 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfmv\" (UniqueName: \"kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.740399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.721517 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:13.783866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.774994 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.824634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.793361 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfmv\" (UniqueName: \"kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv\") pod \"payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:13.970479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.970443 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" event={"ID":"e3ec8754-0b60-4957-9b3f-08f42a4e55bb","Type":"ContainerStarted","Data":"1aedb8526a241fdd16ba36f1f352769783dc0d7cc476f83175fcf9434f70026d"} Oct 30 13:21:13.978958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:13.978927 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck29m\" (UniqueName: \"kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:14.019534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.019503 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:14.057736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.057029 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:21:14.057736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.057678 5693 patch_prober.go:29] interesting pod/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:21:14.057736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.057732 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 13:21:14.153207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.152815 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:14.357216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.357186 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j"] Oct 30 13:21:14.359237 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:14.359207 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35f78d8_c673_49c8_85a1_42433b5f0bcc.slice/crio-395bd8988816208cdae0c010ebc414a72d46323ae9fc46fd6bfe9d58073ba778 WatchSource:0}: Error finding container 395bd8988816208cdae0c010ebc414a72d46323ae9fc46fd6bfe9d58073ba778: Status 404 returned error can't find the container with id 395bd8988816208cdae0c010ebc414a72d46323ae9fc46fd6bfe9d58073ba778 Oct 30 13:21:14.467229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.467204 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c"] Oct 30 13:21:14.469339 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:14.469313 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3fd420_758a_4ac5_9aa1_b268a982e2de.slice/crio-0369002f0a5668251f2ebcd3df1cd6859b390dac5df8ecade9767b2d99589deb WatchSource:0}: Error finding container 0369002f0a5668251f2ebcd3df1cd6859b390dac5df8ecade9767b2d99589deb: Status 404 returned error can't find the container with id 0369002f0a5668251f2ebcd3df1cd6859b390dac5df8ecade9767b2d99589deb Oct 30 13:21:14.635622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.634875 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:14.979216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.979177 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" event={"ID":"d35f78d8-c673-49c8-85a1-42433b5f0bcc","Type":"ContainerStarted","Data":"3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4"} Oct 30 13:21:14.979758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.979242 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" event={"ID":"d35f78d8-c673-49c8-85a1-42433b5f0bcc","Type":"ContainerStarted","Data":"395bd8988816208cdae0c010ebc414a72d46323ae9fc46fd6bfe9d58073ba778"} Oct 30 13:21:14.979758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.979337 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:14.980874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.980849 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" event={"ID":"dc3fd420-758a-4ac5-9aa1-b268a982e2de","Type":"ContainerStarted","Data":"1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460"} Oct 30 13:21:14.981013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.980889 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" event={"ID":"dc3fd420-758a-4ac5-9aa1-b268a982e2de","Type":"ContainerStarted","Data":"0369002f0a5668251f2ebcd3df1cd6859b390dac5df8ecade9767b2d99589deb"} Oct 30 13:21:14.981013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:14.980937 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:15.020091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:15.020042 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" podStartSLOduration=2.020027976 podStartE2EDuration="2.020027976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:15.017177821 +0000 UTC m=+1050.883796005" watchObservedRunningTime="2026-10-30 13:21:15.020027976 +0000 UTC m=+1050.886646185" Oct 30 13:21:15.053055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:15.053014 5693 patch_prober.go:29] interesting pod/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:21:15.053218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:15.053069 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 13:21:15.075820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:15.075770 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" podStartSLOduration=2.075754108 podStartE2EDuration="2.075754108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:15.074667287 +0000 UTC m=+1050.941285479" watchObservedRunningTime="2026-10-30 13:21:15.075754108 +0000 UTC m=+1050.942372280" Oct 30 13:21:16.053622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.053065 5693 patch_prober.go:29] interesting pod/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:21:16.053622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.053135 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 13:21:16.054212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.053652 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" podUID="ae8bd06d-82de-46bc-9592-21699f8fb529" containerName="payload-processing" containerID="cri-o://9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4" gracePeriod=25 Oct 30 13:21:16.077004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.076953 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" podUID="2d79bd41-09af-4f3a-be24-b1512652335d" containerName="payload-pre-processing" containerID="cri-o://c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4" gracePeriod=25 Oct 30 13:21:16.321754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.321673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.653228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.653200 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:21:16.689108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.689080 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:21:16.758380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.758347 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume\") pod \"ae8bd06d-82de-46bc-9592-21699f8fb529\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " Oct 30 13:21:16.759101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.758971 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume" pod "ae8bd06d-82de-46bc-9592-21699f8fb529" (UID: "ae8bd06d-82de-46bc-9592-21699f8fb529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:16.759235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.759187 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl\" (UniqueName: \"kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl\") pod \"ae8bd06d-82de-46bc-9592-21699f8fb529\" (UID: \"ae8bd06d-82de-46bc-9592-21699f8fb529\") " Oct 30 13:21:16.759298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.759263 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n\" (UniqueName: \"kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n\") pod \"2d79bd41-09af-4f3a-be24-b1512652335d\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " Oct 30 13:21:16.760186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.759371 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume\") pod \"2d79bd41-09af-4f3a-be24-b1512652335d\" (UID: \"2d79bd41-09af-4f3a-be24-b1512652335d\") " Oct 30 13:21:16.760510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.760492 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8bd06d-82de-46bc-9592-21699f8fb529-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.760683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.760656 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume" pod "2d79bd41-09af-4f3a-be24-b1512652335d" (UID: "2d79bd41-09af-4f3a-be24-b1512652335d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:16.762907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.762878 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl" pod "ae8bd06d-82de-46bc-9592-21699f8fb529" (UID: "ae8bd06d-82de-46bc-9592-21699f8fb529"). InnerVolumeSpecName "kube-api-access-d4gsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:16.762988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.762915 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n" pod "2d79bd41-09af-4f3a-be24-b1512652335d" (UID: "2d79bd41-09af-4f3a-be24-b1512652335d"). InnerVolumeSpecName "kube-api-access-twr7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:16.862370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.862290 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4gsl\" (UniqueName: \"kubernetes.io/projected/ae8bd06d-82de-46bc-9592-21699f8fb529-kube-api-access-d4gsl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.862370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.862319 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twr7n\" (UniqueName: \"kubernetes.io/projected/2d79bd41-09af-4f3a-be24-b1512652335d-kube-api-access-twr7n\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.862370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.862329 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d79bd41-09af-4f3a-be24-b1512652335d-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.997739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.997694 5693 generic.go:356] "Generic (PLEG): container finished" podID="2d79bd41-09af-4f3a-be24-b1512652335d" containerID="c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4" exitCode=0 Oct 30 13:21:16.997953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.997753 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" Oct 30 13:21:16.997953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.997761 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" event={"ID":"2d79bd41-09af-4f3a-be24-b1512652335d","Type":"ContainerDied","Data":"c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4"} Oct 30 13:21:16.997953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.997829 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2" event={"ID":"2d79bd41-09af-4f3a-be24-b1512652335d","Type":"ContainerDied","Data":"ad0e3f53745dc9fcb828fb2d04ba5c580f70297541f32f78442171066f82fdb8"} Oct 30 13:21:16.997953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.997859 5693 scope.go:122] "RemoveContainer" containerID="c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4" Oct 30 13:21:16.999501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.999471 5693 generic.go:356] "Generic (PLEG): container finished" podID="ae8bd06d-82de-46bc-9592-21699f8fb529" containerID="9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4" exitCode=0 Oct 30 13:21:16.999659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.999539 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" Oct 30 13:21:16.999659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.999538 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" event={"ID":"ae8bd06d-82de-46bc-9592-21699f8fb529","Type":"ContainerDied","Data":"9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4"} Oct 30 13:21:16.999791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:16.999699 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65" event={"ID":"ae8bd06d-82de-46bc-9592-21699f8fb529","Type":"ContainerDied","Data":"52ce1276d7287f3dbbff7ab29e355d3624d810e6010330bcc658af27663db073"} Oct 30 13:21:17.022876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.022852 5693 scope.go:122] "RemoveContainer" containerID="c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4" Oct 30 13:21:17.023185 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:17.023163 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4\": container with ID starting with c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4 not found: ID does not exist" containerID="c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4" Oct 30 13:21:17.023270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.023192 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4"} err="failed to get container status \"c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4\": rpc error: code = NotFound desc = could not find container \"c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4\": container with ID starting with c1622e2bdac1b607b2d0d7956f6c3c4cad2f4109c671a6275460b13fe5581bf4 not found: ID does not exist" Oct 30 13:21:17.023270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.023210 5693 scope.go:122] "RemoveContainer" containerID="9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4" Oct 30 13:21:17.053657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.052028 5693 patch_prober.go:29] interesting pod/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:21:17.053657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.052089 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 13:21:17.062581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.062438 5693 scope.go:122] "RemoveContainer" containerID="9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4" Oct 30 13:21:17.062836 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:17.062796 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4\": container with ID starting with 9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4 not found: ID does not exist" containerID="9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4" Oct 30 13:21:17.062897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.062846 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4"} err="failed to get container status \"9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4\": rpc error: code = NotFound desc = could not find container \"9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4\": container with ID starting with 9c0090cfebed7316a32ace5f14ee87953bacaeebe8f557c5253b4a56a17744f4 not found: ID does not exist" Oct 30 13:21:17.078711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.078687 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65"] Oct 30 13:21:17.089577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.089555 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-78b457cf4-czf65"] Oct 30 13:21:17.123347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.123275 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2"] Oct 30 13:21:17.126575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.126550 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-5455b94f79-gmlc2"] Oct 30 13:21:17.712284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.712250 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:21:17.712464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.712444 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:21:17.855505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.850098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.969101 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq"] Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.970340 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2d79bd41-09af-4f3a-be24-b1512652335d" containerName="payload-pre-processing" Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.971653 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d79bd41-09af-4f3a-be24-b1512652335d" containerName="payload-pre-processing" Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.971737 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ae8bd06d-82de-46bc-9592-21699f8fb529" containerName="payload-processing" Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.971750 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8bd06d-82de-46bc-9592-21699f8fb529" containerName="payload-processing" Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.971966 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d79bd41-09af-4f3a-be24-b1512652335d" containerName="payload-pre-processing" Oct 30 13:21:17.975688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:17.971995 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8bd06d-82de-46bc-9592-21699f8fb529" containerName="payload-processing" Oct 30 13:21:18.053644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.040466 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq"] Oct 30 13:21:18.053644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.040661 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.106623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.105231 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxhw\" (UniqueName: \"kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.106623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.105321 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.106623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.105456 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.106623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.105516 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.115848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.109679 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b7b93a3-serving-cert\"" Oct 30 13:21:18.115848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.109786 5693 patch_prober.go:29] interesting pod/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:21:18.115848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.109828 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 13:21:18.115848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.111542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:18.182853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.182819 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2"] Oct 30 13:21:18.183041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.183013 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" containerID="cri-o://246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76" gracePeriod=30 Oct 30 13:21:18.206959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.206927 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxhw\" (UniqueName: \"kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.207117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.206990 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.207117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.207099 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.207195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.207149 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.207830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.207803 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.209550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.209530 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.209645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.209562 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.363671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.362634 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxhw\" (UniqueName: \"kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw\") pod \"maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.367616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.365015 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d79bd41-09af-4f3a-be24-b1512652335d" path="/var/lib/kubelet/pods/2d79bd41-09af-4f3a-be24-b1512652335d/volumes" Oct 30 13:21:18.367616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.365948 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8bd06d-82de-46bc-9592-21699f8fb529" path="/var/lib/kubelet/pods/ae8bd06d-82de-46bc-9592-21699f8fb529/volumes" Oct 30 13:21:18.378631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.378373 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:18.494621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.490073 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:18.967341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:18.967306 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq"] Oct 30 13:21:18.969982 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:18.969954 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1327ca_babd_4931_8e22_5e7c41778b12.slice/crio-9b537c3afa67dac41e90ceb01a4f26e22cff2d2a9e746459fbde42075edafa4a WatchSource:0}: Error finding container 9b537c3afa67dac41e90ceb01a4f26e22cff2d2a9e746459fbde42075edafa4a: Status 404 returned error can't find the container with id 9b537c3afa67dac41e90ceb01a4f26e22cff2d2a9e746459fbde42075edafa4a Oct 30 13:21:19.101673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:19.090799 5693 patch_prober.go:29] interesting pod/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:21:19.101673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:19.090921 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 13:21:19.177458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:19.177422 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" event={"ID":"8d1327ca-babd-4931-8e22-5e7c41778b12","Type":"ContainerStarted","Data":"9b537c3afa67dac41e90ceb01a4f26e22cff2d2a9e746459fbde42075edafa4a"} Oct 30 13:21:20.041203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:20.041172 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:21:20.041735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:20.041713 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:21:20.042384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:20.042364 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:21:20.191497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:20.191457 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" event={"ID":"8d1327ca-babd-4931-8e22-5e7c41778b12","Type":"ContainerStarted","Data":"0b5d1e9de6082a14f233c0a4f52ff61817630991e45165de101ee0eb1f2bde62"} Oct 30 13:21:20.205449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:20.205419 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:20.233291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:20.233245 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" podStartSLOduration=2.892040093 podStartE2EDuration="3.233227788s" totalImagesPullingTime="341.187695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:20.230793758 +0000 UTC m=+1056.097411950" watchObservedRunningTime="2026-10-30 13:21:20.233227788 +0000 UTC m=+1056.099846021" Oct 30 13:21:21.200193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.200160 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:21.853299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.853266 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn"] Oct 30 13:21:21.923621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.923000 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn"] Oct 30 13:21:21.923621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.923419 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.016229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.994806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:22.016229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.996221 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.016229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.996321 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.016229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.996411 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf56\" (UniqueName: \"kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.016229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:21.996518 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.017271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.017225 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7c0e28-serving-cert\"" Oct 30 13:21:22.098479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.098447 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.098704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.098488 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.098704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.098534 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf56\" (UniqueName: \"kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.098704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.098571 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.099256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.099230 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.101407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.101381 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.101673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.101656 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.118369 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.118306 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf56\" (UniqueName: \"kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56\") pod \"maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.174646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.174581 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:22.358620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.330778 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:22.359254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.339082 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:22.791715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:22.791686 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn"] Oct 30 13:21:22.793294 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:22.793266 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2040a9fd_b253_4415_8952_ad73875ecb6a.slice/crio-8d73012ab3db4f04f1313319618c91795d820e71d5c1d90f568e13d16752d448 WatchSource:0}: Error finding container 8d73012ab3db4f04f1313319618c91795d820e71d5c1d90f568e13d16752d448: Status 404 returned error can't find the container with id 8d73012ab3db4f04f1313319618c91795d820e71d5c1d90f568e13d16752d448 Oct 30 13:21:23.101266 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:23.101175 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:21:23.365977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.346915 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" event={"ID":"2040a9fd-b253-4415-8952-ad73875ecb6a","Type":"ContainerStarted","Data":"8d73012ab3db4f04f1313319618c91795d820e71d5c1d90f568e13d16752d448"} Oct 30 13:21:23.611390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.611356 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:21:23.699661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699565 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.699832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699668 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.699832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699794 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.699971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699836 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:23.699971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699878 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.699971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699925 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.699971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699955 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.700273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.699985 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.700273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.700034 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.700273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.700129 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:23.700438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.700389 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data\") pod \"2c2f529b-d023-440d-8fd1-42351821cde3\" (UID: \"2c2f529b-d023-440d-8fd1-42351821cde3\") " Oct 30 13:21:23.700438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.700408 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:23.701083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.700720 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:23.701215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.701045 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:23.701518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.701494 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.701668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.701522 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.701668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.701537 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.701668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.701551 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.701668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.701565 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2f529b-d023-440d-8fd1-42351821cde3-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.702810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.702789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:23.702887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.702820 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "kube-api-access-w97jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:23.703339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.703314 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:23.703386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.703344 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy" pod "2c2f529b-d023-440d-8fd1-42351821cde3" (UID: "2c2f529b-d023-440d-8fd1-42351821cde3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:23.803561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.803522 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w97jn\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-kube-api-access-w97jn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.803561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.803555 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2f529b-d023-440d-8fd1-42351821cde3-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.803561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.803567 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2f529b-d023-440d-8fd1-42351821cde3-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:23.803874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:23.803578 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2f529b-d023-440d-8fd1-42351821cde3-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:24.198538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.198509 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:24.357672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.357635 5693 generic.go:356] "Generic (PLEG): container finished" podID="2c2f529b-d023-440d-8fd1-42351821cde3" containerID="246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76" exitCode=0 Oct 30 13:21:24.357862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.357687 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" Oct 30 13:21:24.357862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.357721 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" event={"ID":"2c2f529b-d023-440d-8fd1-42351821cde3","Type":"ContainerDied","Data":"246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76"} Oct 30 13:21:24.357862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.357778 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2" event={"ID":"2c2f529b-d023-440d-8fd1-42351821cde3","Type":"ContainerDied","Data":"94a1b79c7746c1577522252f9db3efb12802c052d27ff0b6d80bf4fd952764a0"} Oct 30 13:21:24.357862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.357808 5693 scope.go:122] "RemoveContainer" containerID="246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76" Oct 30 13:21:24.359446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.359424 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" event={"ID":"2040a9fd-b253-4415-8952-ad73875ecb6a","Type":"ContainerStarted","Data":"5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43"} Oct 30 13:21:24.359562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.359469 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:24.368533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.368517 5693 scope.go:122] "RemoveContainer" containerID="93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f" Oct 30 13:21:24.378455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.378434 5693 scope.go:122] "RemoveContainer" containerID="246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76" Oct 30 13:21:24.378775 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:24.378758 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76\": container with ID starting with 246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76 not found: ID does not exist" containerID="246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76" Oct 30 13:21:24.378840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.378783 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76"} err="failed to get container status \"246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76\": rpc error: code = NotFound desc = could not find container \"246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76\": container with ID starting with 246d012463761c4963c0a9e42a9dcd4bc1daf7ad65e260aa3f8219256f521d76 not found: ID does not exist" Oct 30 13:21:24.378840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.378804 5693 scope.go:122] "RemoveContainer" containerID="93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f" Oct 30 13:21:24.379056 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:24.379035 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f\": container with ID starting with 93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f not found: ID does not exist" containerID="93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f" Oct 30 13:21:24.379123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.379063 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f"} err="failed to get container status \"93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f\": rpc error: code = NotFound desc = could not find container \"93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f\": container with ID starting with 93552fe1fce781f723fbf8a6dbcf0f2e863e2bacb54a4bce141d692a2548e14f not found: ID does not exist" Oct 30 13:21:24.421923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.421894 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2"] Oct 30 13:21:24.436919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.436880 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ed395-openshift-default-9d75f778c-bzbj2"] Oct 30 13:21:24.464080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:24.464024 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" podStartSLOduration=3.092082458 podStartE2EDuration="3.464008632s" totalImagesPullingTime="371.926174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:24.462243878 +0000 UTC m=+1060.328862074" watchObservedRunningTime="2026-10-30 13:21:24.464008632 +0000 UTC m=+1060.330626808" Oct 30 13:21:25.571563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:25.571519 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:25.955113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:25.954992 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:26.018418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:26.018387 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:21:26.021572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:26.021543 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:21:26.255817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:26.255776 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" path="/var/lib/kubelet/pods/2c2f529b-d023-440d-8fd1-42351821cde3/volumes" Oct 30 13:21:27.236804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:27.236774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:27.427773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:27.427736 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:21:27.508166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:27.508082 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:28.047489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.047454 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz"] Oct 30 13:21:28.053367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.048854 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" Oct 30 13:21:28.053367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.052142 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" Oct 30 13:21:28.053367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.052436 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" Oct 30 13:21:28.231433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.231403 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.232087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.232059 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz"] Oct 30 13:21:28.246412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.246387 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8ac28e-dockercfg-q8f2q\"" Oct 30 13:21:28.258216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.258194 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8ac28e\"" Oct 30 13:21:28.294024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.293996 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.294398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.294376 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr888\" (UniqueName: \"kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.396362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.396271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr888\" (UniqueName: \"kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.396531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.396375 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.397081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.397059 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.416121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.416095 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr888\" (UniqueName: \"kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888\") pod \"payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.551677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.551643 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:28.602867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.589738 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:28.973094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:28.973068 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz"] Oct 30 13:21:28.975212 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:28.975176 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb17be3b_dd0d_4bd5_8c60_daae72a3ebb5.slice/crio-07aae48b7bcca8cdf6cb9d614751be73cba6227b6ecc308ece043b2c154483e5 WatchSource:0}: Error finding container 07aae48b7bcca8cdf6cb9d614751be73cba6227b6ecc308ece043b2c154483e5: Status 404 returned error can't find the container with id 07aae48b7bcca8cdf6cb9d614751be73cba6227b6ecc308ece043b2c154483e5 Oct 30 13:21:29.456925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:29.456876 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" event={"ID":"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5","Type":"ContainerStarted","Data":"07aae48b7bcca8cdf6cb9d614751be73cba6227b6ecc308ece043b2c154483e5"} Oct 30 13:21:30.305779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:30.305743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:30.388434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:30.387147 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:21:31.481230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:31.481196 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" event={"ID":"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5","Type":"ContainerStarted","Data":"016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9"} Oct 30 13:21:31.481704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:31.481295 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:31.518918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:31.518860 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" podStartSLOduration=2.8375028540000002 podStartE2EDuration="4.518846389s" totalImagesPullingTime="1.681343535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:31.517891932 +0000 UTC m=+1067.384510115" watchObservedRunningTime="2026-10-30 13:21:31.518846389 +0000 UTC m=+1067.385464561" Oct 30 13:21:32.047069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:32.047035 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:32.910041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:32.910003 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:33.294482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.294437 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q"] Oct 30 13:21:33.295322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.295301 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" Oct 30 13:21:33.295322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.295323 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" Oct 30 13:21:33.295566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.295552 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2f529b-d023-440d-8fd1-42351821cde3" containerName="istio-proxy" Oct 30 13:21:33.299412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.299387 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.315716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.315670 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5dcf84-openshift-default-dockercfg-59pq6\"" Oct 30 13:21:33.334546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.334509 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q"] Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441073 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441183 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441252 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441323 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441354 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441525 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hbf\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441559 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441679 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.447650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.441719 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.474664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.473172 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.546991 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547083 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547110 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hbf\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547322 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547384 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547421 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547475 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.547565 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.549683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.548082 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.563756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.562777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.563756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.563065 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.564087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.564062 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.581875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.564587 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.589624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.587381 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.593618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.590143 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.609623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.603753 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hbf\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.609623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.604466 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token\") pod \"e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.633621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.631125 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:33.872589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:33.872533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:34.096355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.096325 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q"] Oct 30 13:21:34.098754 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:34.098728 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad06448_7087_4e86_868c_6ccc0ecf709c.slice/crio-043754f14e3c4f6db9b4fc71eee8ff3dfd93f6f34a06951372933dc5b110a555 WatchSource:0}: Error finding container 043754f14e3c4f6db9b4fc71eee8ff3dfd93f6f34a06951372933dc5b110a555: Status 404 returned error can't find the container with id 043754f14e3c4f6db9b4fc71eee8ff3dfd93f6f34a06951372933dc5b110a555 Oct 30 13:21:34.101343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.101305 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:34.101436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.101377 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:34.101436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.101411 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:34.170634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.167047 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:34.555129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.555075 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" event={"ID":"cad06448-7087-4e86-868c-6ccc0ecf709c","Type":"ContainerStarted","Data":"8b280be808674f3220f3453da6a32ea46dd8fa7c71c312a1984d6a74d5d29b02"} Oct 30 13:21:34.555304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.555154 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" event={"ID":"cad06448-7087-4e86-868c-6ccc0ecf709c","Type":"ContainerStarted","Data":"043754f14e3c4f6db9b4fc71eee8ff3dfd93f6f34a06951372933dc5b110a555"} Oct 30 13:21:34.601688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.601590 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podStartSLOduration=1.60156963 podStartE2EDuration="1.60156963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:34.600580389 +0000 UTC m=+1070.467198562" watchObservedRunningTime="2026-10-30 13:21:34.60156963 +0000 UTC m=+1070.468187825" Oct 30 13:21:34.632039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:34.632011 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:35.637820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:35.637785 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:35.814752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:35.814717 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:36.570669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:36.570636 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:36.571761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:36.571738 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:21:37.153715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:37.153678 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:37.392119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:37.392082 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:37.992618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:37.992238 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:39.075779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:39.075742 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:39.224933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:39.224878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:41.087687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:41.082784 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:21:41.102859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:41.102802 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:21:41.203985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:41.203951 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:41.797821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:41.796756 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:21:42.099678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:42.099488 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:21:42.490270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:42.490239 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:21:43.236452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.236387 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:43.395409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.395367 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:43.615661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.615535 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c"] Oct 30 13:21:43.621965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.621942 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.646409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.646377 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c"] Oct 30 13:21:43.652171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.652147 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1f486a-openshift-default-dockercfg-k9tkh\"" Oct 30 13:21:43.700937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.700904 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.700960 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701043 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701071 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701105 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701271 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701355 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701400 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.701517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.701479 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.724778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.724749 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816255 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816323 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816379 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816453 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816501 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816538 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816577 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816783 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.818742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.816840 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.857623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.821064 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.858058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.826942 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.858283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.826945 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.858396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.827706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.858484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.827988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.911772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.888253 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.912020 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.903450 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.912134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.903990 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.912230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.904465 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr\") pod \"e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:43.924626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.924123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:43.948621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:43.945878 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:44.445814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:44.445787 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c"] Oct 30 13:21:44.448193 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:44.448161 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129df7be_7639_4b69_a599_2f45f8670f08.slice/crio-aa95bba61ed636a95d7230ba500aaf218a5ea2fb5a90c145f75fdb8f57b98036 WatchSource:0}: Error finding container aa95bba61ed636a95d7230ba500aaf218a5ea2fb5a90c145f75fdb8f57b98036: Status 404 returned error can't find the container with id aa95bba61ed636a95d7230ba500aaf218a5ea2fb5a90c145f75fdb8f57b98036 Oct 30 13:21:44.451091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:44.451058 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:44.451196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:44.451129 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:44.451196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:44.451156 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:45.029894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.029853 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" event={"ID":"129df7be-7639-4b69-a599-2f45f8670f08","Type":"ContainerStarted","Data":"ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750"} Oct 30 13:21:45.030240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.030215 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" event={"ID":"129df7be-7639-4b69-a599-2f45f8670f08","Type":"ContainerStarted","Data":"aa95bba61ed636a95d7230ba500aaf218a5ea2fb5a90c145f75fdb8f57b98036"} Oct 30 13:21:45.094143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.088614 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podStartSLOduration=2.088579504 podStartE2EDuration="2.088579504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:45.085333387 +0000 UTC m=+1080.951951561" watchObservedRunningTime="2026-10-30 13:21:45.088579504 +0000 UTC m=+1080.955197678" Oct 30 13:21:45.397670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.390289 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7680-43b600"] Oct 30 13:21:45.402041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.402014 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7680-43b600"] Oct 30 13:21:45.402172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.402100 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:45.461672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.458831 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476wg\" (UniqueName: \"kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg\") pod \"test-curl-7680-43b600\" (UID: \"27ec8e11-5238-45f1-b4d0-06461c1a025f\") " pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:45.469650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.465571 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4bkkd\"" Oct 30 13:21:45.469650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.466629 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:45.567474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.567158 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476wg\" (UniqueName: \"kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg\") pod \"test-curl-7680-43b600\" (UID: \"27ec8e11-5238-45f1-b4d0-06461c1a025f\") " pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:45.588238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.588197 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476wg\" (UniqueName: \"kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg\") pod \"test-curl-7680-43b600\" (UID: \"27ec8e11-5238-45f1-b4d0-06461c1a025f\") " pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:45.789047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.789013 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:45.946586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.946553 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:45.951991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:45.951962 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:46.039123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:46.039032 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:46.040094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:46.040073 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:21:46.095082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:46.095049 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7680-43b600"] Oct 30 13:21:46.097176 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:46.097145 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27ec8e11_5238_45f1_b4d0_06461c1a025f.slice/crio-35ce52c4583b83ecd74872f29936f5091c30ee2cae2fa092e39b071bc9fb51b0 WatchSource:0}: Error finding container 35ce52c4583b83ecd74872f29936f5091c30ee2cae2fa092e39b071bc9fb51b0: Status 404 returned error can't find the container with id 35ce52c4583b83ecd74872f29936f5091c30ee2cae2fa092e39b071bc9fb51b0 Oct 30 13:21:47.052128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:47.052049 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7680-43b600" event={"ID":"27ec8e11-5238-45f1-b4d0-06461c1a025f","Type":"ContainerStarted","Data":"35ce52c4583b83ecd74872f29936f5091c30ee2cae2fa092e39b071bc9fb51b0"} Oct 30 13:21:47.100452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:47.100424 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:47.525531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:47.525494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:47.684073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:47.681102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:47.901675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:47.899787 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:49.091345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:49.091311 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:49.105696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:49.105658 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7680-43b600" event={"ID":"27ec8e11-5238-45f1-b4d0-06461c1a025f","Type":"ContainerStarted","Data":"b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838"} Oct 30 13:21:49.146210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:49.146159 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7680-43b600" podStartSLOduration=1.721041374 podStartE2EDuration="4.146143449s" totalImagesPullingTime="2.425102075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:49.142496981 +0000 UTC m=+1085.009115173" watchObservedRunningTime="2026-10-30 13:21:49.146143449 +0000 UTC m=+1085.012761649" Oct 30 13:21:49.260013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:49.259955 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:50.245191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:50.245156 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7680-43b600"] Oct 30 13:21:50.263085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:50.263049 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7680-43b600"] Oct 30 13:21:50.313415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:50.313383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:51.119638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:51.119565 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7680-43b600" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" containerName="test-curl-7680-43b600" containerID="cri-o://b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838" gracePeriod=2 Oct 30 13:21:51.132835 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:51.132802 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-43b600\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:51.906567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:51.906501 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:52.058381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:52.058346 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:52.217745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:52.217073 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:52.996023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:52.995988 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:53.059697 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:53.059660 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:21:53.379016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.378983 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:53.410639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.410570 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg\" (UniqueName: \"kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg\") pod \"27ec8e11-5238-45f1-b4d0-06461c1a025f\" (UID: \"27ec8e11-5238-45f1-b4d0-06461c1a025f\") " Oct 30 13:21:53.412691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.412664 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg" pod "27ec8e11-5238-45f1-b4d0-06461c1a025f" (UID: "27ec8e11-5238-45f1-b4d0-06461c1a025f"). InnerVolumeSpecName "kube-api-access-476wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:53.512583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.512496 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-476wg\" (UniqueName: \"kubernetes.io/projected/27ec8e11-5238-45f1-b4d0-06461c1a025f-kube-api-access-476wg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:21:53.526069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.526040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:53.668669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.668636 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n"] Oct 30 13:21:53.669387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.669368 5693 cpu_manager.go:414] "containerMap: removing container" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" containerName="test-curl-7680-43b600" Oct 30 13:21:53.669387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.669384 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" containerName="test-curl-7680-43b600" Oct 30 13:21:53.669634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.669520 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" containerName="test-curl-7680-43b600" Oct 30 13:21:53.671015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.670989 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:53.673412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.673393 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:53.690203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.690176 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n"] Oct 30 13:21:53.802684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.791221 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-121d75\"" Oct 30 13:21:53.802684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.791319 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-121d75-dockercfg-ql59n\"" Oct 30 13:21:53.869620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.866842 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzg5x\" (UniqueName: \"kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x\") pod \"payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:53.869620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.867012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:53.990091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.987870 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:53.991979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:53.991939 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzg5x\" (UniqueName: \"kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x\") pod \"payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:54.015486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.015450 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:54.087809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.087730 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzg5x\" (UniqueName: \"kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x\") pod \"payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:54.146642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.146579 5693 generic.go:356] "Generic (PLEG): container finished" podID="27ec8e11-5238-45f1-b4d0-06461c1a025f" containerID="b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838" exitCode=137 Oct 30 13:21:54.146642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.146640 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:54.146886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.146695 5693 scope.go:122] "RemoveContainer" containerID="b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838" Oct 30 13:21:54.157078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.157061 5693 scope.go:122] "RemoveContainer" containerID="b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838" Oct 30 13:21:54.157337 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:54.157320 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838\": container with ID starting with b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838 not found: ID does not exist" containerID="b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838" Oct 30 13:21:54.157437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.157346 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838"} err="failed to get container status \"b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838\": rpc error: code = NotFound desc = could not find container \"b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838\": container with ID starting with b0bab4d35cd50a091e9ed3e80e5a25d713bad9b4284f3ee114ee94f3a9d8f838 not found: ID does not exist" Oct 30 13:21:54.169426 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:54.169395 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-43b600\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:54.258244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.258205 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" path="/var/lib/kubelet/pods/27ec8e11-5238-45f1-b4d0-06461c1a025f/volumes" Oct 30 13:21:54.264464 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:54.264423 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-43b600\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="27ec8e11-5238-45f1-b4d0-06461c1a025f" pod="opendatahub/test-curl-7680-43b600" Oct 30 13:21:54.290856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.290828 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:54.517667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.514022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:54.595118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:54.595024 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n"] Oct 30 13:21:54.598994 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:54.598933 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c0a77c_66d0_4fb0_b6bb_45508f3ebef9.slice/crio-9ba9bee882da510ed8f14c61ba135a018d3cbc2423d4ae9d1855f420aacdde6e WatchSource:0}: Error finding container 9ba9bee882da510ed8f14c61ba135a018d3cbc2423d4ae9d1855f420aacdde6e: Status 404 returned error can't find the container with id 9ba9bee882da510ed8f14c61ba135a018d3cbc2423d4ae9d1855f420aacdde6e Oct 30 13:21:55.155330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:55.155292 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" event={"ID":"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9","Type":"ContainerStarted","Data":"12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7"} Oct 30 13:21:55.155846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:55.155345 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" event={"ID":"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9","Type":"ContainerStarted","Data":"9ba9bee882da510ed8f14c61ba135a018d3cbc2423d4ae9d1855f420aacdde6e"} Oct 30 13:21:55.155846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:55.155389 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:21:55.195416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:55.195345 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" podStartSLOduration=2.195328627 podStartE2EDuration="2.195328627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:55.191734249 +0000 UTC m=+1091.058352425" watchObservedRunningTime="2026-10-30 13:21:55.195328627 +0000 UTC m=+1091.061946802" Oct 30 13:21:56.212056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:56.211973 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:57.850986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.850951 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc"] Oct 30 13:21:57.856060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.856037 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.874004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.873979 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc"] Oct 30 13:21:57.881911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.881884 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5dcf84-serving-cert\"" Oct 30 13:21:57.891956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.891926 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.892093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.892069 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.892150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.892138 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.892277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.892260 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cc5x\" (UniqueName: \"kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.980150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.980115 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:57.993953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.993919 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.994131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.994063 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.994201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.994172 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.994253 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:57.994218 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-serving-cert: secret "maas-api-e2e-ipp-a-5dcf84-serving-cert" not found Oct 30 13:21:57.994317 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:21:57.994300 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls podName:a8a88d50-bfee-45d5-a1b2-0d7dedde5853 nodeName:}" failed. No retries permitted until 2026-10-30 13:21:58.494275406 +0000 UTC m=+1094.360893571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls") pod "maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" (UID: "a8a88d50-bfee-45d5-a1b2-0d7dedde5853") : secret "maas-api-e2e-ipp-a-5dcf84-serving-cert" not found Oct 30 13:21:57.994387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.994366 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cc5x\" (UniqueName: \"kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.994694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.994674 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:57.996733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:57.996712 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:58.016472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:58.016447 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cc5x\" (UniqueName: \"kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:58.111964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:58.111883 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:58.278356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:58.278319 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:58.511390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:58.511343 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:58.513929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:58.513906 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:58.771116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:58.771019 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:21:59.162689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:59.154078 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc"] Oct 30 13:21:59.179642 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:21:59.179368 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a88d50_bfee_45d5_a1b2_0d7dedde5853.slice/crio-8fdf007e28c07255a3e2f5ca02ed845f472c121d5df2addd69271d5deed2be62 WatchSource:0}: Error finding container 8fdf007e28c07255a3e2f5ca02ed845f472c121d5df2addd69271d5deed2be62: Status 404 returned error can't find the container with id 8fdf007e28c07255a3e2f5ca02ed845f472c121d5df2addd69271d5deed2be62 Oct 30 13:21:59.296168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:59.296134 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" event={"ID":"a8a88d50-bfee-45d5-a1b2-0d7dedde5853","Type":"ContainerStarted","Data":"8fdf007e28c07255a3e2f5ca02ed845f472c121d5df2addd69271d5deed2be62"} Oct 30 13:21:59.705093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:21:59.705058 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:00.305312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.305277 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" event={"ID":"a8a88d50-bfee-45d5-a1b2-0d7dedde5853","Type":"ContainerStarted","Data":"38515da18833456b09557a463c7943f8e61e22fcddfe030366eda28280df53de"} Oct 30 13:22:00.305755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.305339 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:22:00.344934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.344885 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" podStartSLOduration=2.954463493 podStartE2EDuration="3.344871189s" totalImagesPullingTime="390.407696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:00.34393705 +0000 UTC m=+1096.210555224" watchObservedRunningTime="2026-10-30 13:22:00.344871189 +0000 UTC m=+1096.211489362" Oct 30 13:22:00.525623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.521120 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d"] Oct 30 13:22:00.537628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.536187 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d"] Oct 30 13:22:00.537628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.536309 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.562912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.561615 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b7b93a3-openshift-default-dockercfg-42stm\"" Oct 30 13:22:00.606651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.606562 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.608081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.608031 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.608462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.608440 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.608718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.608701 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.609062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.609036 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.609287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.609266 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnpp\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.609527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.609506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.609753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.609736 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.609932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.609906 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.711842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.711805 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.711866 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnpp\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.711955 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712034 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712072 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712167 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712203 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712310 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712355 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712447 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712482 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712751 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.712902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.712877 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.713496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.713470 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.714448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.714424 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.714750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.714730 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.731150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.731121 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.731242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.731173 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnpp\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.905561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.905481 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:00.977001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:00.976968 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j"] Oct 30 13:22:01.001881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.001849 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c"] Oct 30 13:22:01.049202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.049118 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq"] Oct 30 13:22:01.049736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.049700 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" podUID="8d1327ca-babd-4931-8e22-5e7c41778b12" containerName="maas-api" containerID="cri-o://0b5d1e9de6082a14f233c0a4f52ff61817630991e45165de101ee0eb1f2bde62" gracePeriod=30 Oct 30 13:22:01.143142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.143105 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.229084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.229051 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d"] Oct 30 13:22:01.230962 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:01.230933 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e6a0e5_95ab_4b20_a36f_ed2c683ffae6.slice/crio-f628b97cfcd70ae84640ed4ede42ce45a548d961befc0273114ee890adc79ccc WatchSource:0}: Error finding container f628b97cfcd70ae84640ed4ede42ce45a548d961befc0273114ee890adc79ccc: Status 404 returned error can't find the container with id f628b97cfcd70ae84640ed4ede42ce45a548d961befc0273114ee890adc79ccc Oct 30 13:22:01.233748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.233704 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:01.233837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.233779 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:01.233837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.233807 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:01.283485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.282355 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.318106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.317840 5693 generic.go:356] "Generic (PLEG): container finished" podID="8d1327ca-babd-4931-8e22-5e7c41778b12" containerID="0b5d1e9de6082a14f233c0a4f52ff61817630991e45165de101ee0eb1f2bde62" exitCode=0 Oct 30 13:22:01.318106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.318029 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" event={"ID":"8d1327ca-babd-4931-8e22-5e7c41778b12","Type":"ContainerDied","Data":"0b5d1e9de6082a14f233c0a4f52ff61817630991e45165de101ee0eb1f2bde62"} Oct 30 13:22:01.324662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.324221 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" event={"ID":"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6","Type":"ContainerStarted","Data":"f628b97cfcd70ae84640ed4ede42ce45a548d961befc0273114ee890adc79ccc"} Oct 30 13:22:01.342126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.342100 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:22:01.424499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.424454 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw\" (UniqueName: \"kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw\") pod \"8d1327ca-babd-4931-8e22-5e7c41778b12\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " Oct 30 13:22:01.424686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.424534 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls\") pod \"8d1327ca-babd-4931-8e22-5e7c41778b12\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " Oct 30 13:22:01.424686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.424584 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs\") pod \"8d1327ca-babd-4931-8e22-5e7c41778b12\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " Oct 30 13:22:01.424686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.424652 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca\") pod \"8d1327ca-babd-4931-8e22-5e7c41778b12\" (UID: \"8d1327ca-babd-4931-8e22-5e7c41778b12\") " Oct 30 13:22:01.425838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.425806 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca" pod "8d1327ca-babd-4931-8e22-5e7c41778b12" (UID: "8d1327ca-babd-4931-8e22-5e7c41778b12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:01.427125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.427094 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw" pod "8d1327ca-babd-4931-8e22-5e7c41778b12" (UID: "8d1327ca-babd-4931-8e22-5e7c41778b12"). InnerVolumeSpecName "kube-api-access-zvxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:01.429084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.429061 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs" pod "8d1327ca-babd-4931-8e22-5e7c41778b12" (UID: "8d1327ca-babd-4931-8e22-5e7c41778b12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:22:01.429329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.429308 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls" pod "8d1327ca-babd-4931-8e22-5e7c41778b12" (UID: "8d1327ca-babd-4931-8e22-5e7c41778b12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:22:01.550325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.550203 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxhw\" (UniqueName: \"kubernetes.io/projected/8d1327ca-babd-4931-8e22-5e7c41778b12-kube-api-access-zvxhw\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:01.550325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.550245 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:01.550325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.550259 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1327ca-babd-4931-8e22-5e7c41778b12-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:01.550325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.550293 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1327ca-babd-4931-8e22-5e7c41778b12-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:01.595796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.595750 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4"] Oct 30 13:22:01.660622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.629049 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w"] Oct 30 13:22:01.800379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.799064 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.932240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:01.932208 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:02.121728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.116042 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:02.393677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.388526 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" event={"ID":"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6","Type":"ContainerStarted","Data":"25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7"} Oct 30 13:22:02.408493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.408456 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" event={"ID":"8d1327ca-babd-4931-8e22-5e7c41778b12","Type":"ContainerDied","Data":"9b537c3afa67dac41e90ceb01a4f26e22cff2d2a9e746459fbde42075edafa4a"} Oct 30 13:22:02.408747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.408731 5693 scope.go:122] "RemoveContainer" containerID="0b5d1e9de6082a14f233c0a4f52ff61817630991e45165de101ee0eb1f2bde62" Oct 30 13:22:02.409022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.409007 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq" Oct 30 13:22:02.472910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.472826 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podStartSLOduration=2.472804925 podStartE2EDuration="2.472804925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:02.465708582 +0000 UTC m=+1098.332326783" watchObservedRunningTime="2026-10-30 13:22:02.472804925 +0000 UTC m=+1098.339423099" Oct 30 13:22:02.537646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.537546 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq"] Oct 30 13:22:02.587906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.587853 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b7b93a3-86f44df-4xnzq"] Oct 30 13:22:02.906166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.906129 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:02.907549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.907525 5693 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:22:02.907655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:02.907574 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:22:03.055935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:03.055877 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:03.209834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:03.207181 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:03.667427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:03.667392 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:03.906182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:03.906127 5693 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:22:03.906401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:03.906243 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:22:04.269318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:04.269284 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1327ca-babd-4931-8e22-5e7c41778b12" path="/var/lib/kubelet/pods/8d1327ca-babd-4931-8e22-5e7c41778b12/volumes" Oct 30 13:22:04.362494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:04.362460 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:04.905905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:04.905872 5693 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:22:04.906365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:04.905941 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:22:05.906117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:05.906087 5693 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:22:05.906527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:05.906137 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:22:05.977899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:05.977858 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" podUID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" containerName="payload-processing" containerID="cri-o://3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4" gracePeriod=25 Oct 30 13:22:05.989652 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:05.989625 5693 logging.go:55] [core] [Channel #1898 SubChannel #1899]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.109:50052", ServerName: "10.134.0.109:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.109:50052: connect: connection refused" Oct 30 13:22:06.002203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.002164 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" podUID="dc3fd420-758a-4ac5-9aa1-b268a982e2de" containerName="payload-pre-processing" containerID="cri-o://1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460" gracePeriod=25 Oct 30 13:22:06.169618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.169513 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:22:06.366511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.366124 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:22:06.379228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.378111 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:22:06.420422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.420338 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:06.469891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.469761 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv\" (UniqueName: \"kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv\") pod \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " Oct 30 13:22:06.469891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.469866 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume\") pod \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\" (UID: \"d35f78d8-c673-49c8-85a1-42433b5f0bcc\") " Oct 30 13:22:06.471790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.471756 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume" pod "d35f78d8-c673-49c8-85a1-42433b5f0bcc" (UID: "d35f78d8-c673-49c8-85a1-42433b5f0bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:06.472346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.472327 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:22:06.473720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.473692 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv" pod "d35f78d8-c673-49c8-85a1-42433b5f0bcc" (UID: "d35f78d8-c673-49c8-85a1-42433b5f0bcc"). InnerVolumeSpecName "kube-api-access-srfmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:06.505899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.505868 5693 generic.go:356] "Generic (PLEG): container finished" podID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" containerID="3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4" exitCode=0 Oct 30 13:22:06.506078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.505981 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" event={"ID":"d35f78d8-c673-49c8-85a1-42433b5f0bcc","Type":"ContainerDied","Data":"3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4"} Oct 30 13:22:06.506078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.506027 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" event={"ID":"d35f78d8-c673-49c8-85a1-42433b5f0bcc","Type":"ContainerDied","Data":"395bd8988816208cdae0c010ebc414a72d46323ae9fc46fd6bfe9d58073ba778"} Oct 30 13:22:06.506078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.506049 5693 scope.go:122] "RemoveContainer" containerID="3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4" Oct 30 13:22:06.506271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.506177 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" Oct 30 13:22:06.510768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.510740 5693 generic.go:356] "Generic (PLEG): container finished" podID="dc3fd420-758a-4ac5-9aa1-b268a982e2de" containerID="1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460" exitCode=0 Oct 30 13:22:06.510935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.510921 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" event={"ID":"dc3fd420-758a-4ac5-9aa1-b268a982e2de","Type":"ContainerDied","Data":"1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460"} Oct 30 13:22:06.510998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.510957 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" event={"ID":"dc3fd420-758a-4ac5-9aa1-b268a982e2de","Type":"ContainerDied","Data":"0369002f0a5668251f2ebcd3df1cd6859b390dac5df8ecade9767b2d99589deb"} Oct 30 13:22:06.511056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.511013 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c" Oct 30 13:22:06.589013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.587703 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srfmv\" (UniqueName: \"kubernetes.io/projected/d35f78d8-c673-49c8-85a1-42433b5f0bcc-kube-api-access-srfmv\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:06.589013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.587736 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f78d8-c673-49c8-85a1-42433b5f0bcc-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:06.610636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.607796 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" podUID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" containerName="payload-processing" containerID="cri-o://5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921" gracePeriod=25 Oct 30 13:22:06.668039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.667784 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" podUID="85675983-0f0d-4528-958b-111d23691544" containerName="payload-pre-processing" containerID="cri-o://757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30" gracePeriod=25 Oct 30 13:22:06.702870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.702612 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume\") pod \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " Oct 30 13:22:06.702870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.702697 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m\" (UniqueName: \"kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m\") pod \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\" (UID: \"dc3fd420-758a-4ac5-9aa1-b268a982e2de\") " Oct 30 13:22:06.703987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.703966 5693 scope.go:122] "RemoveContainer" containerID="3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4" Oct 30 13:22:06.706098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.706056 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume" pod "dc3fd420-758a-4ac5-9aa1-b268a982e2de" (UID: "dc3fd420-758a-4ac5-9aa1-b268a982e2de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:06.706257 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:22:06.706151 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4\": container with ID starting with 3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4 not found: ID does not exist" containerID="3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4" Oct 30 13:22:06.706409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.706383 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4"} err="failed to get container status \"3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4\": rpc error: code = NotFound desc = could not find container \"3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4\": container with ID starting with 3fafa2a54fde19a3d577cbdbf92565bf4702bb6024424644e332c6659d80cbe4 not found: ID does not exist" Oct 30 13:22:06.706540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.706527 5693 scope.go:122] "RemoveContainer" containerID="1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460" Oct 30 13:22:06.710330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.710298 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m" pod "dc3fd420-758a-4ac5-9aa1-b268a982e2de" (UID: "dc3fd420-758a-4ac5-9aa1-b268a982e2de"). InnerVolumeSpecName "kube-api-access-ck29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:06.763432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.763390 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j"] Oct 30 13:22:06.764896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.764868 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j"] Oct 30 13:22:06.805444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.805406 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fd420-758a-4ac5-9aa1-b268a982e2de-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:06.805444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.805446 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck29m\" (UniqueName: \"kubernetes.io/projected/dc3fd420-758a-4ac5-9aa1-b268a982e2de-kube-api-access-ck29m\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:06.815351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.815330 5693 scope.go:122] "RemoveContainer" containerID="1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460" Oct 30 13:22:06.815755 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:22:06.815733 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460\": container with ID starting with 1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460 not found: ID does not exist" containerID="1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460" Oct 30 13:22:06.815912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.815764 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460"} err="failed to get container status \"1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460\": rpc error: code = NotFound desc = could not find container \"1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460\": container with ID starting with 1e387c66dd4e9a3a189fcf2f8627d2c2f9bd6e93e788197bce2888efec101460 not found: ID does not exist" Oct 30 13:22:06.870084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.870045 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c"] Oct 30 13:22:06.876643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.876609 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-wlv9c"] Oct 30 13:22:06.908273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.907257 5693 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:22:06.908273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.907305 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:22:06.989618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:06.989527 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b7b93a3-5f6b9fbf54-v4j7j" podUID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.109:50052\" within 1s: context deadline exceeded" Oct 30 13:22:07.081256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.080938 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:22:07.112402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.112362 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n\" (UniqueName: \"kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n\") pod \"85675983-0f0d-4528-958b-111d23691544\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " Oct 30 13:22:07.112553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.112460 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume\") pod \"85675983-0f0d-4528-958b-111d23691544\" (UID: \"85675983-0f0d-4528-958b-111d23691544\") " Oct 30 13:22:07.112990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.112966 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume" pod "85675983-0f0d-4528-958b-111d23691544" (UID: "85675983-0f0d-4528-958b-111d23691544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:07.113194 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.113163 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85675983-0f0d-4528-958b-111d23691544-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:07.115440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.115397 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n" pod "85675983-0f0d-4528-958b-111d23691544" (UID: "85675983-0f0d-4528-958b-111d23691544"). InnerVolumeSpecName "kube-api-access-jx55n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:07.165394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.165366 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:22:07.214215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.214133 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx\" (UniqueName: \"kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx\") pod \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " Oct 30 13:22:07.214432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.214410 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume\") pod \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\" (UID: \"d7127f9b-61cc-45c8-8fdd-4aecc651e307\") " Oct 30 13:22:07.214881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.214849 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume" pod "d7127f9b-61cc-45c8-8fdd-4aecc651e307" (UID: "d7127f9b-61cc-45c8-8fdd-4aecc651e307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:07.215159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.215134 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7127f9b-61cc-45c8-8fdd-4aecc651e307-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:07.215242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.215165 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx55n\" (UniqueName: \"kubernetes.io/projected/85675983-0f0d-4528-958b-111d23691544-kube-api-access-jx55n\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:07.216664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.216638 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx" pod "d7127f9b-61cc-45c8-8fdd-4aecc651e307" (UID: "d7127f9b-61cc-45c8-8fdd-4aecc651e307"). InnerVolumeSpecName "kube-api-access-9lbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:07.316342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.316312 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lbrx\" (UniqueName: \"kubernetes.io/projected/d7127f9b-61cc-45c8-8fdd-4aecc651e307-kube-api-access-9lbrx\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:07.520080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.520042 5693 generic.go:356] "Generic (PLEG): container finished" podID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" containerID="5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921" exitCode=0 Oct 30 13:22:07.520253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.520089 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" Oct 30 13:22:07.520253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.520126 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" event={"ID":"d7127f9b-61cc-45c8-8fdd-4aecc651e307","Type":"ContainerDied","Data":"5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921"} Oct 30 13:22:07.520253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.520175 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" event={"ID":"d7127f9b-61cc-45c8-8fdd-4aecc651e307","Type":"ContainerDied","Data":"8939dfa5a4140b3d0c4b1f61b514cf3d1d5cb904cd448b8886edf17767373d7f"} Oct 30 13:22:07.520253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.520199 5693 scope.go:122] "RemoveContainer" containerID="5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921" Oct 30 13:22:07.522668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.522638 5693 generic.go:356] "Generic (PLEG): container finished" podID="85675983-0f0d-4528-958b-111d23691544" containerID="757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30" exitCode=0 Oct 30 13:22:07.522668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.522659 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" Oct 30 13:22:07.522855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.522761 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" event={"ID":"85675983-0f0d-4528-958b-111d23691544","Type":"ContainerDied","Data":"757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30"} Oct 30 13:22:07.522855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.522805 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" event={"ID":"85675983-0f0d-4528-958b-111d23691544","Type":"ContainerDied","Data":"dfa9609fec68fc13b374b98d01487b5431740bfad9d46145c29fa00f082ea171"} Oct 30 13:22:07.557689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.557610 5693 scope.go:122] "RemoveContainer" containerID="5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921" Oct 30 13:22:07.558081 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:22:07.558046 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921\": container with ID starting with 5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921 not found: ID does not exist" containerID="5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921" Oct 30 13:22:07.558261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.558217 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921"} err="failed to get container status \"5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921\": rpc error: code = NotFound desc = could not find container \"5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921\": container with ID starting with 5fa5f657b1ace0aa145366272e3b86a3c6c272a4ab8e3055fc8d9e191563b921 not found: ID does not exist" Oct 30 13:22:07.558354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.558338 5693 scope.go:122] "RemoveContainer" containerID="757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30" Oct 30 13:22:07.570576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.570558 5693 scope.go:122] "RemoveContainer" containerID="757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30" Oct 30 13:22:07.570924 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:22:07.570905 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30\": container with ID starting with 757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30 not found: ID does not exist" containerID="757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30" Oct 30 13:22:07.570979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.570932 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30"} err="failed to get container status \"757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30\": rpc error: code = NotFound desc = could not find container \"757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30\": container with ID starting with 757db29620bb3ab1f8a5c233b00794b6c64affe2cc50d6eef0c77323fdae9e30 not found: ID does not exist" Oct 30 13:22:07.597496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.597463 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4"] Oct 30 13:22:07.601320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.601299 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4"] Oct 30 13:22:07.655432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.655389 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w"] Oct 30 13:22:07.658662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.658640 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w"] Oct 30 13:22:07.906278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.906188 5693 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:22:07.906278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:07.906256 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:22:08.061124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.061071 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c2202-5b97b585ff-drg6w" podUID="85675983-0f0d-4528-958b-111d23691544" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.102:50052\" within 1s: context deadline exceeded" Oct 30 13:22:08.061532 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:08.061192 5693 logging.go:55] [core] [Channel #1905 SubChannel #1906]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.102:50052", ServerName: "10.134.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.102:50052: operation was canceled" Oct 30 13:22:08.062142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.062111 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-3c2202-68746d69bb-slrj4" podUID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.103:50052\" within 1s: context deadline exceeded" Oct 30 13:22:08.062263 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:08.062202 5693 logging.go:55] [core] [Channel #1907 SubChannel #1908]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.103:50052", ServerName: "10.134.0.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.103:50052: operation was canceled" Oct 30 13:22:08.068262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.068236 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm"] Oct 30 13:22:08.069288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069267 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" containerName="payload-processing" Oct 30 13:22:08.069288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069285 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" containerName="payload-processing" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069307 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8d1327ca-babd-4931-8e22-5e7c41778b12" containerName="maas-api" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069312 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1327ca-babd-4931-8e22-5e7c41778b12" containerName="maas-api" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069320 5693 cpu_manager.go:414] "containerMap: removing container" podUID="85675983-0f0d-4528-958b-111d23691544" containerName="payload-pre-processing" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069326 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85675983-0f0d-4528-958b-111d23691544" containerName="payload-pre-processing" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069333 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" containerName="payload-processing" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069338 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" containerName="payload-processing" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069347 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dc3fd420-758a-4ac5-9aa1-b268a982e2de" containerName="payload-pre-processing" Oct 30 13:22:08.069415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069351 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3fd420-758a-4ac5-9aa1-b268a982e2de" containerName="payload-pre-processing" Oct 30 13:22:08.069708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069471 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" containerName="payload-processing" Oct 30 13:22:08.069708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069481 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1327ca-babd-4931-8e22-5e7c41778b12" containerName="maas-api" Oct 30 13:22:08.069708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069488 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="85675983-0f0d-4528-958b-111d23691544" containerName="payload-pre-processing" Oct 30 13:22:08.069708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069494 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" containerName="payload-processing" Oct 30 13:22:08.069708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.069500 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3fd420-758a-4ac5-9aa1-b268a982e2de" containerName="payload-pre-processing" Oct 30 13:22:08.074634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.074614 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.099102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.099080 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1f486a-serving-cert\"" Oct 30 13:22:08.104225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.104202 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm"] Oct 30 13:22:08.129482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.129442 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.129697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.129495 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gv5j\" (UniqueName: \"kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.129697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.129586 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.129951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.129925 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.231205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.231175 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.231397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.231353 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.231397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.231387 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gv5j\" (UniqueName: \"kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.231530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.231472 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.232071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.232047 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.233867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.233840 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.233972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.233951 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.260018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.259986 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85675983-0f0d-4528-958b-111d23691544" path="/var/lib/kubelet/pods/85675983-0f0d-4528-958b-111d23691544/volumes" Oct 30 13:22:08.260462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.260443 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35f78d8-c673-49c8-85a1-42433b5f0bcc" path="/var/lib/kubelet/pods/d35f78d8-c673-49c8-85a1-42433b5f0bcc/volumes" Oct 30 13:22:08.260898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.260883 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7127f9b-61cc-45c8-8fdd-4aecc651e307" path="/var/lib/kubelet/pods/d7127f9b-61cc-45c8-8fdd-4aecc651e307/volumes" Oct 30 13:22:08.261267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.261255 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3fd420-758a-4ac5-9aa1-b268a982e2de" path="/var/lib/kubelet/pods/dc3fd420-758a-4ac5-9aa1-b268a982e2de/volumes" Oct 30 13:22:08.262321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.262296 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gv5j\" (UniqueName: \"kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j\") pod \"maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.281019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.280993 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:08.399621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.395012 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:08.420969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.420907 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:08.722441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.721868 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:08.833803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:08.833773 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm"] Oct 30 13:22:08.835512 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:08.835454 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96da0066_a10e_4b12_9284_8635da91d592.slice/crio-b4f6d0a8b25fede2aacc6484c3726bbca1a478e761801d167a61886b0173a78f WatchSource:0}: Error finding container b4f6d0a8b25fede2aacc6484c3726bbca1a478e761801d167a61886b0173a78f: Status 404 returned error can't find the container with id b4f6d0a8b25fede2aacc6484c3726bbca1a478e761801d167a61886b0173a78f Oct 30 13:22:09.218417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.218334 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:09.219260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.219241 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:09.220919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.220106 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:09.604144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.603974 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" event={"ID":"96da0066-a10e-4b12-9284-8635da91d592","Type":"ContainerStarted","Data":"c6eaa386651749a18440b4dd6a52eeae42b120998c002399f160442ace67e7ad"} Oct 30 13:22:09.604144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.604049 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" event={"ID":"96da0066-a10e-4b12-9284-8635da91d592","Type":"ContainerStarted","Data":"b4f6d0a8b25fede2aacc6484c3726bbca1a478e761801d167a61886b0173a78f"} Oct 30 13:22:09.604144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.604099 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:09.647620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:09.645881 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" podStartSLOduration=1.24267992 podStartE2EDuration="1.645861563s" totalImagesPullingTime="403.181643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:09.642900132 +0000 UTC m=+1105.509518331" watchObservedRunningTime="2026-10-30 13:22:09.645861563 +0000 UTC m=+1105.512479758" Oct 30 13:22:10.391746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:10.391717 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:12.447945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:12.447914 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:13.917266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:13.917229 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc"] Oct 30 13:22:13.922775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:13.922751 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:13.942383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:13.942356 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc"] Oct 30 13:22:13.948917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:13.948889 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-121d75-serving-cert\"" Oct 30 13:22:14.025536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.025496 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.025766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.025744 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pd8\" (UniqueName: \"kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.025834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.025792 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.026012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.025993 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.045745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.045707 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:14.127929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.127889 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.128938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.128902 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.157630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.129361 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pd8\" (UniqueName: \"kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.158930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.158900 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.159382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.132446 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.168495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.164331 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pd8\" (UniqueName: \"kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.168495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.165089 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.168495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.165818 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs\") pod \"maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.185592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.183190 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:14.251623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.247967 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:14.323996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.323960 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:14.587463 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.587439 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc"] Oct 30 13:22:14.589389 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:14.589355 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4896d66c_cb07_443e_98c4_07282f59f232.slice/crio-e814910b5dfb40276f0569e1d3eb0ad3c391286b9cfc1b1080baa58231eb9acd WatchSource:0}: Error finding container e814910b5dfb40276f0569e1d3eb0ad3c391286b9cfc1b1080baa58231eb9acd: Status 404 returned error can't find the container with id e814910b5dfb40276f0569e1d3eb0ad3c391286b9cfc1b1080baa58231eb9acd Oct 30 13:22:14.635939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:14.635904 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" event={"ID":"4896d66c-cb07-443e-98c4-07282f59f232","Type":"ContainerStarted","Data":"e814910b5dfb40276f0569e1d3eb0ad3c391286b9cfc1b1080baa58231eb9acd"} Oct 30 13:22:15.611878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:15.611848 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:22:15.643997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:15.643957 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" event={"ID":"4896d66c-cb07-443e-98c4-07282f59f232","Type":"ContainerStarted","Data":"9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b"} Oct 30 13:22:15.644179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:15.644029 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:15.718254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:15.718185 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" podStartSLOduration=2.257803708 podStartE2EDuration="2.718161364s" totalImagesPullingTime="460.357656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:15.715898221 +0000 UTC m=+1111.582516395" watchObservedRunningTime="2026-10-30 13:22:15.718161364 +0000 UTC m=+1111.584779552" Oct 30 13:22:16.136235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:16.136199 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:17.827154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:17.827118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:19.525625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:19.523830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:21.687886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:21.687853 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:22:21.849624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:21.841154 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:23.092426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:23.092383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:23.190477 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:22:23.190437 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:22:23.272108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:23.272069 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:24.320994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:24.320963 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:26.096989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:26.096953 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:28.010382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:28.007476 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:29.708696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:29.708654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:30.126940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:30.126897 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:30.265048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:30.265014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:31.467618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.467559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:31.928638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.928590 5693 generic.go:356] "Generic (PLEG): container finished" podID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerID="8b280be808674f3220f3453da6a32ea46dd8fa7c71c312a1984d6a74d5d29b02" exitCode=0 Oct 30 13:22:31.928854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.928671 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" event={"ID":"cad06448-7087-4e86-868c-6ccc0ecf709c","Type":"ContainerDied","Data":"8b280be808674f3220f3453da6a32ea46dd8fa7c71c312a1984d6a74d5d29b02"} Oct 30 13:22:31.929140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.929118 5693 scope.go:122] "RemoveContainer" containerID="8b280be808674f3220f3453da6a32ea46dd8fa7c71c312a1984d6a74d5d29b02" Oct 30 13:22:31.930745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.930716 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:31.930818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.930781 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:31.930818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:31.930815 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:32.083822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.083718 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:22:32.083822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.083768 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:22:32.961759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.961723 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" event={"ID":"cad06448-7087-4e86-868c-6ccc0ecf709c","Type":"ContainerStarted","Data":"7e8ef3a94e4b36ff179b0449bd72385d9025dad5c87ce572b10a63935e1a11e0"} Oct 30 13:22:32.963585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.963558 5693 generic.go:356] "Generic (PLEG): container finished" podID="129df7be-7639-4b69-a599-2f45f8670f08" containerID="ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750" exitCode=0 Oct 30 13:22:32.963731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.963624 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" event={"ID":"129df7be-7639-4b69-a599-2f45f8670f08","Type":"ContainerDied","Data":"ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750"} Oct 30 13:22:32.963953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.963939 5693 scope.go:122] "RemoveContainer" containerID="ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750" Oct 30 13:22:32.965480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.965452 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:32.965550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.965519 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:32.965628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:32.965552 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:33.422044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:33.422012 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:33.585625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:33.584051 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:33.647629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:33.641638 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:22:33.647629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:33.644673 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:22:33.647629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:33.644731 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 13:22:34.039260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.038659 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" event={"ID":"129df7be-7639-4b69-a599-2f45f8670f08","Type":"ContainerStarted","Data":"60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7"} Oct 30 13:22:34.338752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.338639 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:34.652098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.652002 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:22:34.652378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.652349 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 13:22:34.949104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.949010 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:22:34.949455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.949433 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:22:34.949628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:34.949581 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:22:35.633018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:35.632977 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:22:35.637646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:35.633579 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 13:22:35.948022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:35.947926 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:22:35.949095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:35.949059 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:22:36.363991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:36.363931 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:36.580827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:36.580790 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:22:36.641184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:36.641087 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:22:36.641808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:36.641777 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 13:22:36.956820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:36.956717 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:22:36.957087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:36.957051 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:22:37.635066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:37.635022 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:22:37.635381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:37.635333 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 13:22:37.951675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:37.951467 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:22:37.951675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:37.951533 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:22:38.421770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:38.421732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:38.636384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:38.635905 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:22:38.636384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:38.635968 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 13:22:38.947048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:38.947002 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:22:38.947364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:38.947324 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:22:39.655739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:39.655705 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:22:39.656892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:39.656871 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:22:39.947370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:39.947281 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:22:39.947370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:39.947339 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:22:40.197796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:40.197709 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:40.950813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:40.950786 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:22:40.951264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:40.951251 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:22:40.952048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:40.952023 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:22:41.866702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:41.866667 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:42.808590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:42.808545 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:43.269267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:43.268722 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:43.600334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:43.596835 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:43.767730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:43.767676 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.588530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:44.588494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.745657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:44.741265 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:45.611622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:45.604295 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:45.772064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:45.772026 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:45.918878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:45.918791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:46.185291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:46.185202 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:47.520384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:47.520342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:47.731534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:47.731504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.398209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:48.398165 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.544505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:48.544476 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.729628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:49.727510 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:50.097253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:50.097169 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:50.258387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:50.258338 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:51.507430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:51.507397 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:52.947156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:52.947119 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:53.070218 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:22:53.070179 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:22:53.090899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:53.090870 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:53.230503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:53.230468 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm"] Oct 30 13:22:53.230825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:53.230792 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" containerID="cri-o://1aedb8526a241fdd16ba36f1f352769783dc0d7cc476f83175fcf9434f70026d" gracePeriod=30 Oct 30 13:22:53.234382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:53.234352 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:53.387142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:53.386930 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:54.288632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:54.288589 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:54.434158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:54.434128 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:54.550137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:54.550046 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d"] Oct 30 13:22:54.550319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:54.550255 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" containerID="cri-o://25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7" gracePeriod=30 Oct 30 13:22:54.621385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:54.621339 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:54.971444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:54.960149 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:55.145866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.145524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:55.450186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.450149 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:55.658915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.658881 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp"] Oct 30 13:22:55.663576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.663555 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.680835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.680808 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp"] Oct 30 13:22:55.688503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.688478 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1f486a\"" Oct 30 13:22:55.688742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.688477 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1f486a-dockercfg-jjfvc\"" Oct 30 13:22:55.788021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.787990 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlw89\" (UniqueName: \"kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.788206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.788157 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.890274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.890233 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlw89\" (UniqueName: \"kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.890478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.890391 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.891499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.891465 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.909289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.909264 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlw89\" (UniqueName: \"kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:55.937673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:55.937632 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:56.331622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.320932 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:56.332702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.332293 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:56.461047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.461014 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk"] Oct 30 13:22:56.467264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.467137 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk"] Oct 30 13:22:56.467522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.467234 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.470459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.470384 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:56.494868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.494450 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hxk\" (UniqueName: \"kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk\") pod \"payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.494868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.494628 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.601044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.600983 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hxk\" (UniqueName: \"kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk\") pod \"payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.601510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.601471 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.603406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.603370 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.627015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.626935 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hxk\" (UniqueName: \"kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk\") pod \"payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:56.823299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.823273 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp"] Oct 30 13:22:56.825170 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:56.825144 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bb88af_59ff_426d_b070_c9a4e8944e88.slice/crio-4b5080bca166052bcaf0dc50a3580159c04ae1d2d60626a00d55c57378a7a474 WatchSource:0}: Error finding container 4b5080bca166052bcaf0dc50a3580159c04ae1d2d60626a00d55c57378a7a474: Status 404 returned error can't find the container with id 4b5080bca166052bcaf0dc50a3580159c04ae1d2d60626a00d55c57378a7a474 Oct 30 13:22:56.880795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:56.880768 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:57.208709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.208623 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk"] Oct 30 13:22:57.213444 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:22:57.213412 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439e921f_9e85_4bcd_a458_a7de27e0ca70.slice/crio-118ac00c5ed22227d837819b8374b6c49079397d7b30e17a377f01ccbb1eb248 WatchSource:0}: Error finding container 118ac00c5ed22227d837819b8374b6c49079397d7b30e17a377f01ccbb1eb248: Status 404 returned error can't find the container with id 118ac00c5ed22227d837819b8374b6c49079397d7b30e17a377f01ccbb1eb248 Oct 30 13:22:57.463672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.463548 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" event={"ID":"74bb88af-59ff-426d-b070-c9a4e8944e88","Type":"ContainerStarted","Data":"eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3"} Oct 30 13:22:57.463672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.463641 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" event={"ID":"74bb88af-59ff-426d-b070-c9a4e8944e88","Type":"ContainerStarted","Data":"4b5080bca166052bcaf0dc50a3580159c04ae1d2d60626a00d55c57378a7a474"} Oct 30 13:22:57.464202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.463793 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:22:57.465703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.465675 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" event={"ID":"439e921f-9e85-4bcd-a458-a7de27e0ca70","Type":"ContainerStarted","Data":"fb5e00cfbc9828b2d137ba8906106bb9e907a860992c60f02ddafbc9738bf719"} Oct 30 13:22:57.465802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.465719 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" event={"ID":"439e921f-9e85-4bcd-a458-a7de27e0ca70","Type":"ContainerStarted","Data":"118ac00c5ed22227d837819b8374b6c49079397d7b30e17a377f01ccbb1eb248"} Oct 30 13:22:57.465802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.465739 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:22:57.503380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:57.503328 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" podStartSLOduration=2.503316648 podStartE2EDuration="2.503316648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:57.500659537 +0000 UTC m=+1153.367277712" watchObservedRunningTime="2026-10-30 13:22:57.503316648 +0000 UTC m=+1153.369934821" Oct 30 13:22:58.477330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.477295 5693 generic.go:356] "Generic (PLEG): container finished" podID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerID="1aedb8526a241fdd16ba36f1f352769783dc0d7cc476f83175fcf9434f70026d" exitCode=0 Oct 30 13:22:58.477829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.477365 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" event={"ID":"e3ec8754-0b60-4957-9b3f-08f42a4e55bb","Type":"ContainerDied","Data":"1aedb8526a241fdd16ba36f1f352769783dc0d7cc476f83175fcf9434f70026d"} Oct 30 13:22:58.477829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.477439 5693 scope.go:122] "RemoveContainer" containerID="2aacde4f17ec3cf65a8fdba6b3e6d7dafaaf47b23ea259f9584356337292f50b" Oct 30 13:22:58.520904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.520882 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:22:58.548556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.548526 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.548789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.548772 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.548867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.548823 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:58.548926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.548868 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.548982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.548929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.549039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.548978 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.549092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549062 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:58.549092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549070 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.549557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549213 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.549557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549296 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:58.549557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549353 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.549557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549403 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy\") pod \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\" (UID: \"e3ec8754-0b60-4957-9b3f-08f42a4e55bb\") " Oct 30 13:22:58.550383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.549650 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:58.550383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.550295 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:58.550684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.550664 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.550784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.550689 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.550784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.550702 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.550784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.550715 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.551642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.551576 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:22:58.551928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.551907 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "kube-api-access-q8772". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:58.552707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.552685 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:58.552784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.552711 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token" pod "e3ec8754-0b60-4957-9b3f-08f42a4e55bb" (UID: "e3ec8754-0b60-4957-9b3f-08f42a4e55bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:58.565472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.565221 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" podStartSLOduration=3.565204117 podStartE2EDuration="3.565204117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:57.546113537 +0000 UTC m=+1153.412731739" watchObservedRunningTime="2026-10-30 13:22:58.565204117 +0000 UTC m=+1154.431822292" Oct 30 13:22:58.653107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.653065 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.653107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.653104 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.653357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.653122 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8772\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-kube-api-access-q8772\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.653357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.653137 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.653357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.653150 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ec8754-0b60-4957-9b3f-08f42a4e55bb-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:58.661545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:58.661516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:59.375704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.375671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:59.491078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.491035 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" event={"ID":"e3ec8754-0b60-4957-9b3f-08f42a4e55bb","Type":"ContainerDied","Data":"7824187d4eac1cd61943bd8cbdaaab17e921a697dbd178f9212f719f710fb3ea"} Oct 30 13:22:59.491582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.491153 5693 scope.go:122] "RemoveContainer" containerID="1aedb8526a241fdd16ba36f1f352769783dc0d7cc476f83175fcf9434f70026d" Oct 30 13:22:59.491582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.491351 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm" Oct 30 13:22:59.521804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.521767 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:59.586155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.586085 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm"] Oct 30 13:22:59.589739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.589715 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3c2202-openshift-default-85f59688c7-4chwm"] Oct 30 13:22:59.836324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.836300 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:22:59.876050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876017 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876068 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876132 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876152 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876196 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876286 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876344 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876472 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:59.876621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876557 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.876621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876593 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:59.876842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876803 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:59.876842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876814 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:59.877102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.876850 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket\") pod \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\" (UID: \"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6\") " Oct 30 13:22:59.877183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.877127 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:59.877717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.877694 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.877880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.877721 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.877880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.877735 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.877880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.877746 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.877880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.877757 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.879923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.879898 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:22:59.880358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.880317 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:22:59.880885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.880847 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "kube-api-access-8gnpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:59.881102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.881070 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token" pod "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" (UID: "75e6a0e5-95ab-4b20-a36f-ed2c683ffae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:59.979007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.978974 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnpp\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-kube-api-access-8gnpp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.979007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.979006 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.979223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.979020 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:22:59.979223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:22:59.979031 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:00.092193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.092153 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:00.259488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.259407 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" path="/var/lib/kubelet/pods/e3ec8754-0b60-4957-9b3f-08f42a4e55bb/volumes" Oct 30 13:23:00.504762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.504723 5693 generic.go:356] "Generic (PLEG): container finished" podID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerID="25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7" exitCode=0 Oct 30 13:23:00.505260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.504777 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" event={"ID":"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6","Type":"ContainerDied","Data":"25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7"} Oct 30 13:23:00.505260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.504847 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" event={"ID":"75e6a0e5-95ab-4b20-a36f-ed2c683ffae6","Type":"ContainerDied","Data":"f628b97cfcd70ae84640ed4ede42ce45a548d961befc0273114ee890adc79ccc"} Oct 30 13:23:00.505260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.504876 5693 scope.go:122] "RemoveContainer" containerID="25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7" Oct 30 13:23:00.505530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.505511 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d" Oct 30 13:23:00.518324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.518302 5693 scope.go:122] "RemoveContainer" containerID="25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7" Oct 30 13:23:00.518648 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:00.518591 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7\": container with ID starting with 25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7 not found: ID does not exist" containerID="25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7" Oct 30 13:23:00.518710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.518656 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7"} err="failed to get container status \"25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7\": rpc error: code = NotFound desc = could not find container \"25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7\": container with ID starting with 25dcaeee690c5a154948a15dbd6b4e890f9171173400cbaf707ebc1b77ddf4d7 not found: ID does not exist" Oct 30 13:23:00.558189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.558137 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d"] Oct 30 13:23:00.567662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.567585 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:00.570132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.570107 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-bzk9d"] Oct 30 13:23:00.868674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:00.862344 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:01.018177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.018147 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:01.370574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.370545 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:01.438717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.438682 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9"] Oct 30 13:23:01.439496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.439476 5693 cpu_manager.go:414] "containerMap: removing container" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" Oct 30 13:23:01.439570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.439498 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" Oct 30 13:23:01.439570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.439534 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" Oct 30 13:23:01.439570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.439539 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" Oct 30 13:23:01.439716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.439701 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" Oct 30 13:23:01.439752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.439726 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" containerName="istio-proxy" Oct 30 13:23:01.443459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.443434 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.464805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.464774 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9"] Oct 30 13:23:01.475616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.467992 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c9e946-dockercfg-rnp2n\"" Oct 30 13:23:01.475616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.468054 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c9e946\"" Oct 30 13:23:01.500300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.500269 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln"] Oct 30 13:23:01.501088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.501067 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" Oct 30 13:23:01.501088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.501087 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" Oct 30 13:23:01.501296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.501283 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ec8754-0b60-4957-9b3f-08f42a4e55bb" containerName="istio-proxy" Oct 30 13:23:01.504961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.504933 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:01.520135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.520098 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg\") pod \"payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.520813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.520220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.523295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.523270 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:01.524940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.524920 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln"] Oct 30 13:23:01.686669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.681736 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:01.686669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.681877 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2f2\" (UniqueName: \"kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2\") pod \"payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:01.686669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.681997 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg\") pod \"payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.686669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.682085 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.686669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.683120 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.778627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.762030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg\") pod \"payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:01.815208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.811702 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:01.815584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.815564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2f2\" (UniqueName: \"kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2\") pod \"payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:01.816518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.812580 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:01.912623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:01.900221 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2f2\" (UniqueName: \"kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2\") pod \"payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:02.061641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:02.061587 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:02.126581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:02.126552 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:02.327455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:02.327379 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e6a0e5-95ab-4b20-a36f-ed2c683ffae6" path="/var/lib/kubelet/pods/75e6a0e5-95ab-4b20-a36f-ed2c683ffae6/volumes" Oct 30 13:23:02.576302 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:23:02.576272 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61edd7bd_7327_4486_a019_867f2898c0ff.slice/crio-4e95b022eb5d241cc3109a5840b0891a532c324feebe3c41d7ad10930ccce1db WatchSource:0}: Error finding container 4e95b022eb5d241cc3109a5840b0891a532c324feebe3c41d7ad10930ccce1db: Status 404 returned error can't find the container with id 4e95b022eb5d241cc3109a5840b0891a532c324feebe3c41d7ad10930ccce1db Oct 30 13:23:02.576645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:02.576378 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9"] Oct 30 13:23:02.641082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:02.641041 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:02.732227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:02.717286 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln"] Oct 30 13:23:03.533938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.533902 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" event={"ID":"61edd7bd-7327-4486-a019-867f2898c0ff","Type":"ContainerStarted","Data":"87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330"} Oct 30 13:23:03.534159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.533968 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:03.534159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.533991 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" event={"ID":"61edd7bd-7327-4486-a019-867f2898c0ff","Type":"ContainerStarted","Data":"4e95b022eb5d241cc3109a5840b0891a532c324feebe3c41d7ad10930ccce1db"} Oct 30 13:23:03.535572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.535546 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" event={"ID":"454c01f1-0956-42b7-9ede-7791000dae14","Type":"ContainerStarted","Data":"e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796"} Oct 30 13:23:03.535678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.535586 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" event={"ID":"454c01f1-0956-42b7-9ede-7791000dae14","Type":"ContainerStarted","Data":"5dc98ca46caf4afa423be4e944faafbd5b17a094ce5451bd867971b8e976ec00"} Oct 30 13:23:03.535746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.535733 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:03.584479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.575718 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" podStartSLOduration=2.575699872 podStartE2EDuration="2.575699872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:03.572264624 +0000 UTC m=+1159.438882796" watchObservedRunningTime="2026-10-30 13:23:03.575699872 +0000 UTC m=+1159.442318046" Oct 30 13:23:03.613574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:03.613524 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" podStartSLOduration=2.613509968 podStartE2EDuration="2.613509968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:03.609586985 +0000 UTC m=+1159.476205158" watchObservedRunningTime="2026-10-30 13:23:03.613509968 +0000 UTC m=+1159.480128160" Oct 30 13:23:04.223476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:04.223441 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd"] Oct 30 13:23:04.259167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:04.259134 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh"] Oct 30 13:23:04.416130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:04.416098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:04.560642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:04.560533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:06.128985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:06.128951 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:06.374662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:06.363589 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:06.904172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:06.904101 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:08.479899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:08.479867 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:23:08.480286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:08.479943 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:23:09.249667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.249555 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:09.251259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.249882 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" podUID="687a6f30-a6c8-49c9-ac79-8da51b65e21f" containerName="payload-processing" containerID="cri-o://99ee0112a9b9ff0e8fc20f194ff1f0946fde1d9c5b1a34c6c85891c86d30aeee" gracePeriod=25 Oct 30 13:23:09.269052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.267249 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" podUID="6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" containerName="payload-pre-processing" containerID="cri-o://00d4a4b6040fa64fe1bef31600100eba3706eccd3c5d76325ad63fc0077f503a" gracePeriod=25 Oct 30 13:23:09.611345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.611299 5693 generic.go:356] "Generic (PLEG): container finished" podID="6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" containerID="00d4a4b6040fa64fe1bef31600100eba3706eccd3c5d76325ad63fc0077f503a" exitCode=0 Oct 30 13:23:09.611804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.611391 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" event={"ID":"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4","Type":"ContainerDied","Data":"00d4a4b6040fa64fe1bef31600100eba3706eccd3c5d76325ad63fc0077f503a"} Oct 30 13:23:09.613161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.613137 5693 generic.go:356] "Generic (PLEG): container finished" podID="687a6f30-a6c8-49c9-ac79-8da51b65e21f" containerID="99ee0112a9b9ff0e8fc20f194ff1f0946fde1d9c5b1a34c6c85891c86d30aeee" exitCode=0 Oct 30 13:23:09.613308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.613189 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" event={"ID":"687a6f30-a6c8-49c9-ac79-8da51b65e21f","Type":"ContainerDied","Data":"99ee0112a9b9ff0e8fc20f194ff1f0946fde1d9c5b1a34c6c85891c86d30aeee"} Oct 30 13:23:09.613308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.613234 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" event={"ID":"687a6f30-a6c8-49c9-ac79-8da51b65e21f","Type":"ContainerDied","Data":"346d2d5fe120c000a8e481fbf6b013ac9107408307ae348a6c087d46821c7677"} Oct 30 13:23:09.613308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.613261 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346d2d5fe120c000a8e481fbf6b013ac9107408307ae348a6c087d46821c7677" Oct 30 13:23:09.618775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.618749 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:23:09.640218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.640189 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:23:09.661870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.661834 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume\") pod \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " Oct 30 13:23:09.662037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.662018 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2\" (UniqueName: \"kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2\") pod \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\" (UID: \"687a6f30-a6c8-49c9-ac79-8da51b65e21f\") " Oct 30 13:23:09.662278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.662250 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume" pod "687a6f30-a6c8-49c9-ac79-8da51b65e21f" (UID: "687a6f30-a6c8-49c9-ac79-8da51b65e21f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:09.664206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.663983 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a6f30-a6c8-49c9-ac79-8da51b65e21f-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:09.677252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.677148 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2" pod "687a6f30-a6c8-49c9-ac79-8da51b65e21f" (UID: "687a6f30-a6c8-49c9-ac79-8da51b65e21f"). InnerVolumeSpecName "kube-api-access-q2bm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:09.765180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.765094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m\" (UniqueName: \"kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m\") pod \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " Oct 30 13:23:09.765357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.765318 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume\") pod \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\" (UID: \"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4\") " Oct 30 13:23:09.765790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.765762 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume" pod "6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" (UID: "6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:09.765912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.765898 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bm2\" (UniqueName: \"kubernetes.io/projected/687a6f30-a6c8-49c9-ac79-8da51b65e21f-kube-api-access-q2bm2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:09.765953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.765915 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:09.767328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.767302 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m" pod "6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" (UID: "6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4"). InnerVolumeSpecName "kube-api-access-8zb6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:09.867835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:09.867799 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zb6m\" (UniqueName: \"kubernetes.io/projected/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4-kube-api-access-8zb6m\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:10.622662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.622573 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" Oct 30 13:23:10.623103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.622585 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh" event={"ID":"6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4","Type":"ContainerDied","Data":"1c9fc405951bcb552f1c3d7175f25c88593e1a8457b84dcc586e3963c8e8126f"} Oct 30 13:23:10.623103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.622741 5693 scope.go:122] "RemoveContainer" containerID="00d4a4b6040fa64fe1bef31600100eba3706eccd3c5d76325ad63fc0077f503a" Oct 30 13:23:10.623103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.622804 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd" Oct 30 13:23:10.686706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.686674 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh"] Oct 30 13:23:10.692532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.692502 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c0e28-747bfd5699-gcldh"] Oct 30 13:23:10.730588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.730552 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd"] Oct 30 13:23:10.734144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.734114 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c0e28-7c94f56b58-r6nwd"] Oct 30 13:23:10.910176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.910091 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq"] Oct 30 13:23:10.910950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.910926 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" containerName="payload-pre-processing" Oct 30 13:23:10.911058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.910953 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" containerName="payload-pre-processing" Oct 30 13:23:10.911058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.911000 5693 cpu_manager.go:414] "containerMap: removing container" podUID="687a6f30-a6c8-49c9-ac79-8da51b65e21f" containerName="payload-processing" Oct 30 13:23:10.911058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.911005 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687a6f30-a6c8-49c9-ac79-8da51b65e21f" containerName="payload-processing" Oct 30 13:23:10.911167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.911140 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" containerName="payload-pre-processing" Oct 30 13:23:10.911167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.911153 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="687a6f30-a6c8-49c9-ac79-8da51b65e21f" containerName="payload-processing" Oct 30 13:23:10.916955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.916934 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:10.932440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.932321 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8fb1a8-serving-cert\"" Oct 30 13:23:10.932585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.932474 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq"] Oct 30 13:23:10.984633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.984578 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:10.985277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.985248 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:10.985396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.985289 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srb4p\" (UniqueName: \"kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:10.985454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:10.985429 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.041330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.041296 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.086941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.086905 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.086941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.086943 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srb4p\" (UniqueName: \"kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.087271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.087007 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.087271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.087073 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.087271 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:11.087172 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-serving-cert: secret "maas-api-e2e-api-b-8fb1a8-serving-cert" not found Oct 30 13:23:11.087271 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:11.087249 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls podName:841bf8bd-a836-4214-a788-17025ef8e75e nodeName:}" failed. No retries permitted until 2026-10-30 13:23:11.587228617 +0000 UTC m=+1167.453846782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls") pod "maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" (UID: "841bf8bd-a836-4214-a788-17025ef8e75e") : secret "maas-api-e2e-api-b-8fb1a8-serving-cert" not found Oct 30 13:23:11.087822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.087796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.148625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.145255 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srb4p\" (UniqueName: \"kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.148625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.145827 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.191672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.191096 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.333810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.333779 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.615526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.615483 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.618128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.618104 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\") pod \"maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:11.833257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:11.833228 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:12.178716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:12.178261 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq"] Oct 30 13:23:12.232551 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:23:12.232487 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841bf8bd_a836_4214_a788_17025ef8e75e.slice/crio-2b99f416b1943a437af1ee3881b80276cf8fac9a822a9222f6fff2abc2a55294 WatchSource:0}: Error finding container 2b99f416b1943a437af1ee3881b80276cf8fac9a822a9222f6fff2abc2a55294: Status 404 returned error can't find the container with id 2b99f416b1943a437af1ee3881b80276cf8fac9a822a9222f6fff2abc2a55294 Oct 30 13:23:12.256454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:12.256425 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687a6f30-a6c8-49c9-ac79-8da51b65e21f" path="/var/lib/kubelet/pods/687a6f30-a6c8-49c9-ac79-8da51b65e21f/volumes" Oct 30 13:23:12.256887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:12.256872 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4" path="/var/lib/kubelet/pods/6bfd37c2-e2ce-4bac-ae6d-0c8247b3f7d4/volumes" Oct 30 13:23:12.643583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:12.643544 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" event={"ID":"841bf8bd-a836-4214-a788-17025ef8e75e","Type":"ContainerStarted","Data":"2b99f416b1943a437af1ee3881b80276cf8fac9a822a9222f6fff2abc2a55294"} Oct 30 13:23:12.774843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:12.774807 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:13.653684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:13.653644 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" event={"ID":"841bf8bd-a836-4214-a788-17025ef8e75e","Type":"ContainerStarted","Data":"1880c6526a906744a04e587299e872ca2fd4c1bc2f9b58a92585d098345eeacb"} Oct 30 13:23:13.654075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:13.653850 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:13.695003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:13.694954 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" podStartSLOduration=3.219322967 podStartE2EDuration="3.694936714s" totalImagesPullingTime="475.613747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:13.693082498 +0000 UTC m=+1169.559700671" watchObservedRunningTime="2026-10-30 13:23:13.694936714 +0000 UTC m=+1169.561554886" Oct 30 13:23:14.537634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:14.525945 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:14.554343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:14.554308 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:23:14.558113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:14.558086 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:23:14.560698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:14.560669 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-kg4wx_89c0aafb-9900-4eb3-90ec-d26f746bf51a/payload-processing/0.log" Oct 30 13:23:14.776206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:14.776089 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-kg4wx_89c0aafb-9900-4eb3-90ec-d26f746bf51a/payload-processing/0.log" Oct 30 13:23:14.969253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:14.969173 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk_439e921f-9e85-4bcd-a458-a7de27e0ca70/payload-processing/0.log" Oct 30 13:23:16.216468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:16.216435 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:17.172268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.172227 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:17.254040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.254013 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5"] Oct 30 13:23:17.258986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.258960 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.278091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.278060 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5"] Oct 30 13:23:17.280421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.280373 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8ac28e\"/\"openshift-service-ca.crt\"" Oct 30 13:23:17.280421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.280400 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8ac28e\"/\"kube-root-ca.crt\"" Oct 30 13:23:17.280679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.280561 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8ac28e\"/\"ipp-route-8ac28e-kserve-self-signed-certs\"" Oct 30 13:23:17.285655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.285635 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8ac28e\"/\"default-dockercfg-5pr9c\"" Oct 30 13:23:17.344699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.344642 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:17.359083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.359055 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.359253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.359127 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.359371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.359341 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxnnq\" (UniqueName: \"kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.359444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.359391 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.359851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.359829 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.360028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.360008 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.471740 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.471995 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.472098 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnnq\" (UniqueName: \"kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.472132 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.472288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.472381 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.472936 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.473212 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.487678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.478046 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.521631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.513558 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.532414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.532373 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.556590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.556489 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxnnq\" (UniqueName: \"kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq\") pod \"ipp-route-8ac28e-kserve-7c5c974c49-2njc5\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.581622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.580947 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:17.992265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:17.992239 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5"] Oct 30 13:23:17.994877 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:23:17.994843 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd643a9af_c04e_40b4_b515_843e51324084.slice/crio-ccf32028b078c82b6c5af6617c9283dea52c2c0167c00a8b53641c26b76777b7 WatchSource:0}: Error finding container ccf32028b078c82b6c5af6617c9283dea52c2c0167c00a8b53641c26b76777b7: Status 404 returned error can't find the container with id ccf32028b078c82b6c5af6617c9283dea52c2c0167c00a8b53641c26b76777b7 Oct 30 13:23:18.431079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:18.430987 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:18.744496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:18.744449 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" event={"ID":"d643a9af-c04e-40b4-b515-843e51324084","Type":"ContainerStarted","Data":"239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e"} Oct 30 13:23:18.744749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:18.744502 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" event={"ID":"d643a9af-c04e-40b4-b515-843e51324084","Type":"ContainerStarted","Data":"ccf32028b078c82b6c5af6617c9283dea52c2c0167c00a8b53641c26b76777b7"} Oct 30 13:23:18.786975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:18.786921 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" podStartSLOduration=1.7869074820000002 podStartE2EDuration="1.786907482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:18.783665462 +0000 UTC m=+1174.650283656" watchObservedRunningTime="2026-10-30 13:23:18.786907482 +0000 UTC m=+1174.653525655" Oct 30 13:23:19.583369 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:19.583331 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:19.676002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:19.675975 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:23:20.568923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:20.568895 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:20.704977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:20.704944 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:20.848273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:20.839126 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:22.414453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:22.414419 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:23.060719 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:23.060680 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:23:24.185559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:24.185527 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:25.808359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:25.808211 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.285611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.285570 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq"] Oct 30 13:23:27.290908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.290885 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.305394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.305367 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3a82db-serving-cert\"" Oct 30 13:23:27.309949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.309918 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq"] Oct 30 13:23:27.412220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.412186 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xv2\" (UniqueName: \"kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.412422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.412338 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.412422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.412387 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.412574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.412556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.413341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.413315 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.524630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.520641 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.524630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.520824 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.524630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.520951 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xv2\" (UniqueName: \"kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.524630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.521105 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.524630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.521999 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.585618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.583042 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:27.586116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.586086 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.593216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.592147 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.593216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.592220 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.596444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.594760 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xv2\" (UniqueName: \"kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2\") pod \"maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.615498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.609220 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:27.616090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.615995 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:27.907978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.907890 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:23:27.977784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:27.977750 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq"] Oct 30 13:23:28.906963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:28.906924 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" event={"ID":"4c59269e-5a35-4147-bab4-74aabaef0898","Type":"ContainerStarted","Data":"296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60"} Oct 30 13:23:28.907466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:28.906984 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" event={"ID":"4c59269e-5a35-4147-bab4-74aabaef0898","Type":"ContainerStarted","Data":"c86e0b6ea62eba3ab482f4169147cc70f30b4c97c4ca1ff8dc12f6d15de45a1d"} Oct 30 13:23:28.957618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:28.956492 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" podStartSLOduration=1.585165352 podStartE2EDuration="1.956473635s" totalImagesPullingTime="371.308283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:28.956295929 +0000 UTC m=+1184.822914119" watchObservedRunningTime="2026-10-30 13:23:28.956473635 +0000 UTC m=+1184.823091809" Oct 30 13:23:29.339132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:29.325042 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:29.914657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:29.914609 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:31.169389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:31.157906 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:33.440280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:33.440236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:35.579680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:35.579628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:35.927226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:35.927154 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:23:37.343966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:37.343933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:39.234890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:39.234851 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:39.376010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:39.375961 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:41.056736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:41.056701 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:43.514627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:43.504159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:45.378099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:45.378063 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn"] Oct 30 13:23:45.378537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:45.378336 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" podUID="2040a9fd-b253-4415-8952-ad73875ecb6a" containerName="maas-api" containerID="cri-o://5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43" gracePeriod=30 Oct 30 13:23:45.475420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:45.475387 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:45.905384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:45.905355 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:23:46.007365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.006257 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56\" (UniqueName: \"kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56\") pod \"2040a9fd-b253-4415-8952-ad73875ecb6a\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " Oct 30 13:23:46.007365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.006398 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls\") pod \"2040a9fd-b253-4415-8952-ad73875ecb6a\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " Oct 30 13:23:46.007365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.006446 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca\") pod \"2040a9fd-b253-4415-8952-ad73875ecb6a\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " Oct 30 13:23:46.007365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.006586 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs\") pod \"2040a9fd-b253-4415-8952-ad73875ecb6a\" (UID: \"2040a9fd-b253-4415-8952-ad73875ecb6a\") " Oct 30 13:23:46.018885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.018844 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca" pod "2040a9fd-b253-4415-8952-ad73875ecb6a" (UID: "2040a9fd-b253-4415-8952-ad73875ecb6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:46.025136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.025099 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56" pod "2040a9fd-b253-4415-8952-ad73875ecb6a" (UID: "2040a9fd-b253-4415-8952-ad73875ecb6a"). InnerVolumeSpecName "kube-api-access-vpf56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:46.025433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.025411 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls" pod "2040a9fd-b253-4415-8952-ad73875ecb6a" (UID: "2040a9fd-b253-4415-8952-ad73875ecb6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:46.034569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.034534 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs" pod "2040a9fd-b253-4415-8952-ad73875ecb6a" (UID: "2040a9fd-b253-4415-8952-ad73875ecb6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:46.103096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.102927 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:46.197637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.176496 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf56\" (UniqueName: \"kubernetes.io/projected/2040a9fd-b253-4415-8952-ad73875ecb6a-kube-api-access-vpf56\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:46.197637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.176530 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:46.197637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.176542 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2040a9fd-b253-4415-8952-ad73875ecb6a-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:46.197637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.176552 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2040a9fd-b253-4415-8952-ad73875ecb6a-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:46.231404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.231352 5693 generic.go:356] "Generic (PLEG): container finished" podID="2040a9fd-b253-4415-8952-ad73875ecb6a" containerID="5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43" exitCode=0 Oct 30 13:23:46.231618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.231468 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" Oct 30 13:23:46.231618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.231494 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" event={"ID":"2040a9fd-b253-4415-8952-ad73875ecb6a","Type":"ContainerDied","Data":"5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43"} Oct 30 13:23:46.231618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.231551 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn" event={"ID":"2040a9fd-b253-4415-8952-ad73875ecb6a","Type":"ContainerDied","Data":"8d73012ab3db4f04f1313319618c91795d820e71d5c1d90f568e13d16752d448"} Oct 30 13:23:46.231779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.231574 5693 scope.go:122] "RemoveContainer" containerID="5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43" Oct 30 13:23:46.247508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.247484 5693 scope.go:122] "RemoveContainer" containerID="5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43" Oct 30 13:23:46.247901 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:46.247870 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43\": container with ID starting with 5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43 not found: ID does not exist" containerID="5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43" Oct 30 13:23:46.248008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.247912 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43"} err="failed to get container status \"5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43\": rpc error: code = NotFound desc = could not find container \"5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43\": container with ID starting with 5927c350463d74b4c967a42db6ec8e8ac4f7f021286cfaac8172031939ebde43 not found: ID does not exist" Oct 30 13:23:46.283128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.283094 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn"] Oct 30 13:23:46.286567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.286542 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c0e28-7b6d65fff-tb6qn"] Oct 30 13:23:46.366556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.364318 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:46.501069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.501029 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:46.892785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:46.892758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:47.635465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:47.635441 5693 kubelet.go:1689] "Image garbage collection succeeded" Oct 30 13:23:48.270626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:48.266426 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2040a9fd-b253-4415-8952-ad73875ecb6a" path="/var/lib/kubelet/pods/2040a9fd-b253-4415-8952-ad73875ecb6a/volumes" Oct 30 13:23:48.476343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:48.476311 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:50.204608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:50.204556 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:50.409506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:50.409475 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:50.789379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:50.789340 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n"] Oct 30 13:23:50.913454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:50.913423 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:51.063062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.062952 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:51.293211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.293176 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb"] Oct 30 13:23:51.294181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.294161 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2040a9fd-b253-4415-8952-ad73875ecb6a" containerName="maas-api" Oct 30 13:23:51.294181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.294182 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2040a9fd-b253-4415-8952-ad73875ecb6a" containerName="maas-api" Oct 30 13:23:51.294301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.294291 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2040a9fd-b253-4415-8952-ad73875ecb6a" containerName="maas-api" Oct 30 13:23:51.299981 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.299957 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.314656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.314582 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e3e617-openshift-default-dockercfg-vrtvw\"" Oct 30 13:23:51.331916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.331887 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb"] Oct 30 13:23:51.453624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.452985 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:51.474383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.474313 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.474796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.474757 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ftw\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.497002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.496970 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.497397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.497374 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.497550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.497535 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.497806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.497781 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.497939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.497923 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.498138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.498123 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.498316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.498301 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.609756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.609663 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.610543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.610506 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ftw\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.611224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.611201 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.611788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.611768 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.612917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.612897 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.613340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.613322 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.613838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.613819 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.614425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.611592 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.614555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.613256 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.614817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.610428 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.614817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.612828 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.615101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.615082 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.615570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.615553 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.616130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.615467 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.631853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.631706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ftw\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.635434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.635273 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.690367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.689288 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.690367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.689516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy\") pod \"e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:51.919381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:51.919269 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:52.100146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.100120 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:52.302632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.289076 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:52.317027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.316965 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:23:52.317189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.317080 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:23:52.317189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.317132 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:23:52.350190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.350144 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb"] Oct 30 13:23:52.360344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:52.360313 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" event={"ID":"c466e2e1-ee81-4508-ae95-1c0e56fdef9a","Type":"ContainerStarted","Data":"3b1ea83d0fc0e91c513726cf9f5e6480d22d2aa777addc8cd35872e9aaf9da7a"} Oct 30 13:23:53.058253 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:53.058216 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:23:53.318096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:53.318007 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:53.369149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:53.369108 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" event={"ID":"c466e2e1-ee81-4508-ae95-1c0e56fdef9a","Type":"ContainerStarted","Data":"884e6297f64f7ccd8626b47b74b313623747fbd44341d64ecf1b45f3b043bbb3"} Oct 30 13:23:53.435350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:53.435295 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podStartSLOduration=2.4352810910000002 podStartE2EDuration="2.435281091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:53.420852568 +0000 UTC m=+1209.287470742" watchObservedRunningTime="2026-10-30 13:23:53.435281091 +0000 UTC m=+1209.301899263" Oct 30 13:23:53.814979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:53.814948 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:53.926627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:53.924953 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:53.956620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:53.954099 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:54.001655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.001139 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:54.169310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.169234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:54.323626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.322030 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:54.407626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.404090 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:54.426045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.419681 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:23:54.634621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.631881 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9"] Oct 30 13:23:54.634621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.631963 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d"] Oct 30 13:23:54.634621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.632158 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" containerID="cri-o://620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b" gracePeriod=30 Oct 30 13:23:54.638295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.638251 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq"] Oct 30 13:23:54.638682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.638651 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" containerID="cri-o://e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4" gracePeriod=30 Oct 30 13:23:54.639070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.639019 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" containerID="cri-o://0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba" gracePeriod=30 Oct 30 13:23:54.647282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.647182 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72"] Oct 30 13:23:54.647645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:54.647619 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" containerID="cri-o://521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783" gracePeriod=30 Oct 30 13:23:55.790521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:55.790473 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" podUID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" containerName="payload-pre-processing" containerID="cri-o://12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7" gracePeriod=25 Oct 30 13:23:55.970627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:55.963241 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.170033 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:23:56.169943 5693 logging.go:55] [core] [Channel #2311 SubChannel #2312]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: connect: connection refused" Oct 30 13:23:56.170862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.170818 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.372417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.372393 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:23:56.379811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.379782 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.408237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.408207 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x\" (UniqueName: \"kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x\") pod \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " Oct 30 13:23:56.408454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.408436 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume\") pod \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\" (UID: \"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9\") " Oct 30 13:23:56.408812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.408789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume" pod "c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" (UID: "c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:56.409188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.409165 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:56.410280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.410257 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x" pod "c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" (UID: "c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9"). InnerVolumeSpecName "kube-api-access-tzg5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:56.434989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.434952 5693 generic.go:356] "Generic (PLEG): container finished" podID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" containerID="12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7" exitCode=0 Oct 30 13:23:56.435144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.435001 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" Oct 30 13:23:56.435144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.435033 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" event={"ID":"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9","Type":"ContainerDied","Data":"12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7"} Oct 30 13:23:56.435144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.435114 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" event={"ID":"c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9","Type":"ContainerDied","Data":"9ba9bee882da510ed8f14c61ba135a018d3cbc2423d4ae9d1855f420aacdde6e"} Oct 30 13:23:56.435320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.435145 5693 scope.go:122] "RemoveContainer" containerID="12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7" Oct 30 13:23:56.448983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.448954 5693 scope.go:122] "RemoveContainer" containerID="12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7" Oct 30 13:23:56.449337 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:23:56.449313 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7\": container with ID starting with 12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7 not found: ID does not exist" containerID="12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7" Oct 30 13:23:56.449446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.449344 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7"} err="failed to get container status \"12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7\": rpc error: code = NotFound desc = could not find container \"12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7\": container with ID starting with 12189c5a91b7bd710250228905b20a7e7624c385c00df64eb7f23a514384c0b7 not found: ID does not exist" Oct 30 13:23:56.493945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.493910 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n"] Oct 30 13:23:56.503142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.503105 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n"] Oct 30 13:23:56.511089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.511064 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzg5x\" (UniqueName: \"kubernetes.io/projected/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9-kube-api-access-tzg5x\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:23:56.529814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.529777 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:23:56.529976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.529845 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:23:56.531495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.531468 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.687673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:56.687485 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:57.170122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:57.170081 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-121d75-6d4f4695-8xk9n" podUID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.117:50052\" within 1s: context deadline exceeded" Oct 30 13:23:57.974534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:57.974501 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:58.131235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:58.131200 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:58.279745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:58.279581 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:58.315189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:58.298955 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" path="/var/lib/kubelet/pods/c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9/volumes" Oct 30 13:23:59.953291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:59.953257 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:23:59.984824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:23:59.984796 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:24:00.014910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.014840 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.023720 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.015430 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.023917 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.023986 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024054 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024136 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024487 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024626 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024706 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024739 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024946 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.024992 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.025021 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.025068 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket\") pod \"6b54d4a7-0815-4fa8-958e-67463f66156e\" (UID: \"6b54d4a7-0815-4fa8-958e-67463f66156e\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.025127 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.030061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.025256 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.044278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.044229 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.071622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.067523 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.081427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.076280 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket\") pod \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\" (UID: \"9a1ab878-6a5a-4513-a340-9b3ea31b7d61\") " Oct 30 13:24:00.081427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.077452 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.081427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.077483 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.081427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.077500 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.081427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.081315 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.082482 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.083522 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.083881 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.084338 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.084512 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.084730 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.099173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.085131 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:00.104850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.104767 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "kube-api-access-d9cl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.116715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.116642 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179692 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179724 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179740 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179753 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9cl5\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-kube-api-access-d9cl5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179766 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b54d4a7-0815-4fa8-958e-67463f66156e-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179781 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179795 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179790 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179807 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179820 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179835 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179850 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "kube-api-access-m8p4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.179894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.179889 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.180671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.180056 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token" pod "9a1ab878-6a5a-4513-a340-9b3ea31b7d61" (UID: "9a1ab878-6a5a-4513-a340-9b3ea31b7d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.188916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.185848 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo" pod "6b54d4a7-0815-4fa8-958e-67463f66156e" (UID: "6b54d4a7-0815-4fa8-958e-67463f66156e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:00.281814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.281723 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.281814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.281743 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8p4m\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-kube-api-access-m8p4m\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.281814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.281755 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.281814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.281764 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b54d4a7-0815-4fa8-958e-67463f66156e-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.281814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.281773 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1ab878-6a5a-4513-a340-9b3ea31b7d61-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.375073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.375045 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:24:00.443860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.443765 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:24:00.466628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.466578 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:00.485165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485127 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485260 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485322 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485445 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485539 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.485567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485558 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.485710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485628 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485728 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485787 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485812 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs\") pod \"194f40db-5435-4052-982e-5156e160cd7e\" (UID: \"194f40db-5435-4052-982e-5156e160cd7e\") " Oct 30 13:24:00.485988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485871 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.485988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.485893 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:00.486424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.486396 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.486581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.486556 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.486581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.486582 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.486774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.486611 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194f40db-5435-4052-982e-5156e160cd7e-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.486774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.486624 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.486774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.486635 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.487401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.487372 5693 generic.go:356] "Generic (PLEG): container finished" podID="194f40db-5435-4052-982e-5156e160cd7e" containerID="521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783" exitCode=0 Oct 30 13:24:00.487506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.487447 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" Oct 30 13:24:00.487575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.487453 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" event={"ID":"194f40db-5435-4052-982e-5156e160cd7e","Type":"ContainerDied","Data":"521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783"} Oct 30 13:24:00.487656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.487575 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72" event={"ID":"194f40db-5435-4052-982e-5156e160cd7e","Type":"ContainerDied","Data":"eeddd5f78d832ab81828296f95923722354ce9d4426937f2ff3b2e061532a6ef"} Oct 30 13:24:00.487656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.487634 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.487726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.487655 5693 scope.go:122] "RemoveContainer" containerID="521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783" Oct 30 13:24:00.488127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.488093 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:00.488241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.488218 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "kube-api-access-ph7b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.488731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.488700 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy" pod "194f40db-5435-4052-982e-5156e160cd7e" (UID: "194f40db-5435-4052-982e-5156e160cd7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.489165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.489140 5693 generic.go:356] "Generic (PLEG): container finished" podID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerID="0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba" exitCode=0 Oct 30 13:24:00.489257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.489192 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" Oct 30 13:24:00.489257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.489224 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" event={"ID":"9a1ab878-6a5a-4513-a340-9b3ea31b7d61","Type":"ContainerDied","Data":"0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba"} Oct 30 13:24:00.489368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.489269 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9" event={"ID":"9a1ab878-6a5a-4513-a340-9b3ea31b7d61","Type":"ContainerDied","Data":"15699e58db8848bdae4deb05a1428b5911a0ea3d4d94c1203f8bcdf2fd58e4d0"} Oct 30 13:24:00.491280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.491256 5693 generic.go:356] "Generic (PLEG): container finished" podID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerID="e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4" exitCode=0 Oct 30 13:24:00.491381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.491297 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" Oct 30 13:24:00.491381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.491310 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" event={"ID":"6b54d4a7-0815-4fa8-958e-67463f66156e","Type":"ContainerDied","Data":"e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4"} Oct 30 13:24:00.491381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.491358 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq" event={"ID":"6b54d4a7-0815-4fa8-958e-67463f66156e","Type":"ContainerDied","Data":"697d607c3eff629019df241663988ba1194cdd3c0b082bb7435350b2c6bb3140"} Oct 30 13:24:00.493116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.493095 5693 generic.go:356] "Generic (PLEG): container finished" podID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerID="620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b" exitCode=0 Oct 30 13:24:00.493234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.493138 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" Oct 30 13:24:00.493234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.493164 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" event={"ID":"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6","Type":"ContainerDied","Data":"620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b"} Oct 30 13:24:00.493234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.493202 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d" event={"ID":"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6","Type":"ContainerDied","Data":"00e732e3543b01fc4255dd7733dfea6134d41ae19aa671c19a5ef44688e91bcf"} Oct 30 13:24:00.502832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.502781 5693 scope.go:122] "RemoveContainer" containerID="521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783" Oct 30 13:24:00.511802 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:00.511752 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783\": container with ID starting with 521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783 not found: ID does not exist" containerID="521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783" Oct 30 13:24:00.512212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.512064 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783"} err="failed to get container status \"521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783\": rpc error: code = NotFound desc = could not find container \"521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783\": container with ID starting with 521125674ac51dc074c5fa7d5e7b52488a74087f8c9e8e82a25deefd93d6f783 not found: ID does not exist" Oct 30 13:24:00.514266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.513660 5693 scope.go:122] "RemoveContainer" containerID="0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba" Oct 30 13:24:00.529755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.529485 5693 scope.go:122] "RemoveContainer" containerID="0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba" Oct 30 13:24:00.529956 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:00.529933 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba\": container with ID starting with 0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba not found: ID does not exist" containerID="0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba" Oct 30 13:24:00.530055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.529968 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba"} err="failed to get container status \"0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba\": rpc error: code = NotFound desc = could not find container \"0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba\": container with ID starting with 0b2418d5addf3346495d7862db207d258d488c0fbda6c150c3e1625272cc77ba not found: ID does not exist" Oct 30 13:24:00.530055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.529990 5693 scope.go:122] "RemoveContainer" containerID="e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4" Oct 30 13:24:00.542513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.542273 5693 scope.go:122] "RemoveContainer" containerID="614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4" Oct 30 13:24:00.554305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.554279 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq"] Oct 30 13:24:00.558766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.558746 5693 scope.go:122] "RemoveContainer" containerID="e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4" Oct 30 13:24:00.559092 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:00.559067 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4\": container with ID starting with e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4 not found: ID does not exist" containerID="e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4" Oct 30 13:24:00.559294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.559098 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4"} err="failed to get container status \"e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4\": rpc error: code = NotFound desc = could not find container \"e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4\": container with ID starting with e2704c5ec6256dcee3ec715a2617a5508e38b65cdda4ac72e627e1dfd375baa4 not found: ID does not exist" Oct 30 13:24:00.559294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.559122 5693 scope.go:122] "RemoveContainer" containerID="614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4" Oct 30 13:24:00.559493 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:00.559467 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4\": container with ID starting with 614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4 not found: ID does not exist" containerID="614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4" Oct 30 13:24:00.559562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.559504 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4"} err="failed to get container status \"614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4\": rpc error: code = NotFound desc = could not find container \"614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4\": container with ID starting with 614fad998825d820a03a6341d0d87d248be1980996d31b6856b85ae3350704f4 not found: ID does not exist" Oct 30 13:24:00.559562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.559524 5693 scope.go:122] "RemoveContainer" containerID="620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b" Oct 30 13:24:00.560080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.560006 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lcbnq"] Oct 30 13:24:00.572925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.572904 5693 scope.go:122] "RemoveContainer" containerID="620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b" Oct 30 13:24:00.573259 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:00.573233 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b\": container with ID starting with 620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b not found: ID does not exist" containerID="620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b" Oct 30 13:24:00.573348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.573269 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b"} err="failed to get container status \"620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b\": rpc error: code = NotFound desc = could not find container \"620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b\": container with ID starting with 620a7dfd676828f611fcc955b15349d709ca62bacfa3ed82b8b51c655b66388b not found: ID does not exist" Oct 30 13:24:00.587361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.587332 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.587498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.587472 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.587581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.587563 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.587715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.587692 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.587840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.587759 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.588652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.588545 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.588652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.588625 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.588872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.588854 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.589476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.588919 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.589476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.588967 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.589476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.589041 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket\") pod \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\" (UID: \"0cc74c94-465f-4a25-8e4d-733cd2e3c7c6\") " Oct 30 13:24:00.589476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.589272 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.589476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.589278 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:00.589476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.589426 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.590016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.589993 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590016 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:00.590088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590024 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194f40db-5435-4052-982e-5156e160cd7e-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590038 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590051 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194f40db-5435-4052-982e-5156e160cd7e-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590065 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7b7\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-kube-api-access-ph7b7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590078 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590091 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194f40db-5435-4052-982e-5156e160cd7e-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590103 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590116 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.590582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590555 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "kube-api-access-8ct79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.590879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.590861 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:00.592093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.592076 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token" pod "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" (UID: "0cc74c94-465f-4a25-8e4d-733cd2e3c7c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:00.599766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.599746 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9"] Oct 30 13:24:00.603128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.603107 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-z6pl9"] Oct 30 13:24:00.665661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.665581 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:00.691433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.691403 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ct79\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-kube-api-access-8ct79\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.691433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.691429 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.691640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.691440 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.691640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.691450 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:00.948622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:00.948349 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:01.007773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:01.007640 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d"] Oct 30 13:24:01.033678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:01.033607 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-pwb4d"] Oct 30 13:24:01.046225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:01.046184 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72"] Oct 30 13:24:01.059013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:01.058981 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-7ss72"] Oct 30 13:24:02.256183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.256137 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" path="/var/lib/kubelet/pods/0cc74c94-465f-4a25-8e4d-733cd2e3c7c6/volumes" Oct 30 13:24:02.256797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.256777 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194f40db-5435-4052-982e-5156e160cd7e" path="/var/lib/kubelet/pods/194f40db-5435-4052-982e-5156e160cd7e/volumes" Oct 30 13:24:02.257272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.257257 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" path="/var/lib/kubelet/pods/6b54d4a7-0815-4fa8-958e-67463f66156e/volumes" Oct 30 13:24:02.258055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.258039 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" path="/var/lib/kubelet/pods/9a1ab878-6a5a-4513-a340-9b3ea31b7d61/volumes" Oct 30 13:24:02.299074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.299043 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:02.371198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.371165 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn"] Oct 30 13:24:02.371987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.371962 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" Oct 30 13:24:02.371987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.371982 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372023 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" containerName="payload-pre-processing" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372028 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" containerName="payload-pre-processing" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372046 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372051 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372066 5693 cpu_manager.go:414] "containerMap: removing container" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372071 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372078 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" Oct 30 13:24:02.372151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372083 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" Oct 30 13:24:02.372385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372184 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" Oct 30 13:24:02.372385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372197 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c0a77c-66d0-4fb0-b6bb-45508f3ebef9" containerName="payload-pre-processing" Oct 30 13:24:02.372385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372205 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc74c94-465f-4a25-8e4d-733cd2e3c7c6" containerName="istio-proxy" Oct 30 13:24:02.372385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372213 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="194f40db-5435-4052-982e-5156e160cd7e" containerName="istio-proxy" Oct 30 13:24:02.372385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372219 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1ab878-6a5a-4513-a340-9b3ea31b7d61" containerName="istio-proxy" Oct 30 13:24:02.372385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.372225 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" Oct 30 13:24:02.377133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.377115 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.393732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.393700 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn"] Oct 30 13:24:02.400422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.400398 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5cee02\"" Oct 30 13:24:02.411763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.411728 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5cee02-dockercfg-8dpqb\"" Oct 30 13:24:02.439829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.439790 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj"] Oct 30 13:24:02.442047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.442018 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" Oct 30 13:24:02.442178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.442063 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b54d4a7-0815-4fa8-958e-67463f66156e" containerName="istio-proxy" Oct 30 13:24:02.448457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.445623 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:02.449706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.449680 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.452829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.452800 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.452941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.452889 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcxj\" (UniqueName: \"kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj\") pod \"payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.473417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.473361 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj"] Oct 30 13:24:02.736766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.709528 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.737223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.737190 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.738399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.738372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7b6\" (UniqueName: \"kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6\") pod \"payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.738659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.738636 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcxj\" (UniqueName: \"kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj\") pod \"payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.739071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.727794 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:02.739279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.738247 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.857031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.855127 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.857031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.855319 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7b6\" (UniqueName: \"kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6\") pod \"payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.857649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.857571 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.861967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.860098 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcxj\" (UniqueName: \"kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj\") pod \"payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:02.915660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.912238 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7b6\" (UniqueName: \"kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6\") pod \"payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:02.993561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:02.993475 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:03.160117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.160080 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:03.300704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.300412 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn"] Oct 30 13:24:03.303581 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:24:03.303507 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62743f5d_7cf4_431f_9b73_ee73f11b9d42.slice/crio-7e71b3c51d28b0cdc828a3c0fce475b992bcc98ef79ffdec799494c4428f271a WatchSource:0}: Error finding container 7e71b3c51d28b0cdc828a3c0fce475b992bcc98ef79ffdec799494c4428f271a: Status 404 returned error can't find the container with id 7e71b3c51d28b0cdc828a3c0fce475b992bcc98ef79ffdec799494c4428f271a Oct 30 13:24:03.462983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.462945 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj"] Oct 30 13:24:03.465505 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:24:03.465475 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0653aa_d044_4ba8_9467_2ada5bd41b14.slice/crio-1408eec01b9147f45271ab9a7c5d81793d9ee216b04d2ef02db1652a16b25472 WatchSource:0}: Error finding container 1408eec01b9147f45271ab9a7c5d81793d9ee216b04d2ef02db1652a16b25472: Status 404 returned error can't find the container with id 1408eec01b9147f45271ab9a7c5d81793d9ee216b04d2ef02db1652a16b25472 Oct 30 13:24:03.744691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.744653 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" event={"ID":"fd0653aa-d044-4ba8-9467-2ada5bd41b14","Type":"ContainerStarted","Data":"9f0d325304a2cc2c716b0a666f9613493f3ad14e90f6db9fe5ce745e122294fd"} Oct 30 13:24:03.744890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.744715 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" event={"ID":"fd0653aa-d044-4ba8-9467-2ada5bd41b14","Type":"ContainerStarted","Data":"1408eec01b9147f45271ab9a7c5d81793d9ee216b04d2ef02db1652a16b25472"} Oct 30 13:24:03.744968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.744883 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:03.746559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.746532 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" event={"ID":"62743f5d-7cf4-431f-9b73-ee73f11b9d42","Type":"ContainerStarted","Data":"ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd"} Oct 30 13:24:03.746744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.746570 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" event={"ID":"62743f5d-7cf4-431f-9b73-ee73f11b9d42","Type":"ContainerStarted","Data":"7e71b3c51d28b0cdc828a3c0fce475b992bcc98ef79ffdec799494c4428f271a"} Oct 30 13:24:03.746744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.746590 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:03.787583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.787526 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" podStartSLOduration=1.787511101 podStartE2EDuration="1.787511101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:03.78354945 +0000 UTC m=+1219.650167622" watchObservedRunningTime="2026-10-30 13:24:03.787511101 +0000 UTC m=+1219.654129274" Oct 30 13:24:03.821585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:03.821469 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" podStartSLOduration=1.821438064 podStartE2EDuration="1.821438064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:03.820024069 +0000 UTC m=+1219.686642242" watchObservedRunningTime="2026-10-30 13:24:03.821438064 +0000 UTC m=+1219.688056251" Oct 30 13:24:04.453752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:04.446557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:06.014744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:06.014667 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:07.435225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.435191 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc"] Oct 30 13:24:07.436144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.436011 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" containerID="cri-o://9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56" gracePeriod=30 Oct 30 13:24:07.449005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.448976 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh"] Oct 30 13:24:07.449414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.449379 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" containerID="cri-o://4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978" gracePeriod=30 Oct 30 13:24:07.453906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.453872 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9"] Oct 30 13:24:07.454116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.454088 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" containerID="cri-o://a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e" gracePeriod=30 Oct 30 13:24:07.458345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.458319 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77"] Oct 30 13:24:07.458555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.458527 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" containerID="cri-o://9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10" gracePeriod=30 Oct 30 13:24:07.740684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:07.740655 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:09.937774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:09.937740 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:12.000167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.000131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:12.766473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.766447 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:24:12.804165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.804133 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:24:12.812702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.812666 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:24:12.879655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.879611 5693 generic.go:356] "Generic (PLEG): container finished" podID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerID="9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56" exitCode=0 Oct 30 13:24:12.879899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.879702 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" event={"ID":"d8a0a847-3d24-4406-b285-026bf3db0c77","Type":"ContainerDied","Data":"9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56"} Oct 30 13:24:12.879899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.879763 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" event={"ID":"d8a0a847-3d24-4406-b285-026bf3db0c77","Type":"ContainerDied","Data":"a352a291d0464c3f2da8044817ed12325312381ca9ad28be68ace5fcd81800fc"} Oct 30 13:24:12.879899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.879796 5693 scope.go:122] "RemoveContainer" containerID="9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56" Oct 30 13:24:12.881202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.879929 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc" Oct 30 13:24:12.882006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.881970 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:24:12.885519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.885484 5693 generic.go:356] "Generic (PLEG): container finished" podID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerID="9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10" exitCode=0 Oct 30 13:24:12.885664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.885555 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" event={"ID":"5163624d-38bf-425b-bb38-dd6f1b14668a","Type":"ContainerDied","Data":"9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10"} Oct 30 13:24:12.885744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.885665 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" event={"ID":"5163624d-38bf-425b-bb38-dd6f1b14668a","Type":"ContainerDied","Data":"c0015a3abaf14c317a6d3413253e1ee2331245100e1b538af247a3686ca3354d"} Oct 30 13:24:12.887254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.887231 5693 generic.go:356] "Generic (PLEG): container finished" podID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerID="4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978" exitCode=0 Oct 30 13:24:12.887356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.887290 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" Oct 30 13:24:12.887469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.887439 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" event={"ID":"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93","Type":"ContainerDied","Data":"4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978"} Oct 30 13:24:12.887589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.887490 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh" event={"ID":"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93","Type":"ContainerDied","Data":"db556b45afa4722ad0787caf5f036b5a2fad9eb4c4d51a961f6719445fe23bed"} Oct 30 13:24:12.889229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.889206 5693 generic.go:356] "Generic (PLEG): container finished" podID="055c3603-f114-4fd7-8201-263f8d428598" containerID="a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e" exitCode=0 Oct 30 13:24:12.889331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.889315 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" event={"ID":"055c3603-f114-4fd7-8201-263f8d428598","Type":"ContainerDied","Data":"a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e"} Oct 30 13:24:12.889398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.889350 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" event={"ID":"055c3603-f114-4fd7-8201-263f8d428598","Type":"ContainerDied","Data":"2bbdee490fd4d41548fca5a46fb63675afe604c4dae609fe6d083f742022c6c1"} Oct 30 13:24:12.889452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.889412 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9" Oct 30 13:24:12.892153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892058 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892104 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.892153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892143 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892175 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.892399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892211 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.892399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892253 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892287 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892346 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892412 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892475 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892516 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892547 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892569 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892668 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.892709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892697 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.893071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892740 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.893071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892847 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.893071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892913 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.893071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.892969 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.893071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893020 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.893316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893121 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.893316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893162 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.893316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893219 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.893316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893263 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.893316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893290 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data\") pod \"d8a0a847-3d24-4406-b285-026bf3db0c77\" (UID: \"d8a0a847-3d24-4406-b285-026bf3db0c77\") " Oct 30 13:24:12.893558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893321 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket\") pod \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\" (UID: \"0f2d53dd-aa16-4bc1-8208-5fbdd164ed93\") " Oct 30 13:24:12.893558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.893350 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm\") pod \"055c3603-f114-4fd7-8201-263f8d428598\" (UID: \"055c3603-f114-4fd7-8201-263f8d428598\") " Oct 30 13:24:12.894700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.894677 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.895373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.895343 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.895591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.895571 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.895768 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.895990 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.896072 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.896345 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.896638 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.896967 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.897063 5693 scope.go:122] "RemoveContainer" containerID="9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56" Oct 30 13:24:12.897687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.897389 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.898292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.898150 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.899500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.899092 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "kube-api-access-2bqzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.899500 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:12.899190 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56\": container with ID starting with 9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56 not found: ID does not exist" containerID="9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56" Oct 30 13:24:12.899500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.899223 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56"} err="failed to get container status \"9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56\": rpc error: code = NotFound desc = could not find container \"9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56\": container with ID starting with 9002e3e9b3899ccaa43d35b8625be1700a24d45d34e47dd18fb6b283a61afe56 not found: ID does not exist" Oct 30 13:24:12.899500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.899247 5693 scope.go:122] "RemoveContainer" containerID="9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10" Oct 30 13:24:12.900736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.900107 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.900924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.900886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.901037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901012 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.901132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901107 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.901185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901140 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.901231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901212 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "kube-api-access-9q6gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.901949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901680 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.901949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901742 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "kube-api-access-d6v22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.901949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901895 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.901949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.901932 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo" pod "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" (UID: "0f2d53dd-aa16-4bc1-8208-5fbdd164ed93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:12.903461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.903435 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.903666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.903641 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:12.903900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.903876 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.904064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.904046 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token" pod "055c3603-f114-4fd7-8201-263f8d428598" (UID: "055c3603-f114-4fd7-8201-263f8d428598"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.904797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.904776 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo" pod "d8a0a847-3d24-4406-b285-026bf3db0c77" (UID: "d8a0a847-3d24-4406-b285-026bf3db0c77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:12.921218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.921169 5693 scope.go:122] "RemoveContainer" containerID="9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10" Oct 30 13:24:12.921679 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:12.921622 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10\": container with ID starting with 9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10 not found: ID does not exist" containerID="9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10" Oct 30 13:24:12.921835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.921663 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10"} err="failed to get container status \"9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10\": rpc error: code = NotFound desc = could not find container \"9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10\": container with ID starting with 9c6ce9e22729b1afd830359f8e6bb9181e85cf40bf9f76f5d65085ffd2956f10 not found: ID does not exist" Oct 30 13:24:12.921835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.921699 5693 scope.go:122] "RemoveContainer" containerID="4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978" Oct 30 13:24:12.934231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.934208 5693 scope.go:122] "RemoveContainer" containerID="4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978" Oct 30 13:24:12.934583 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:12.934554 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978\": container with ID starting with 4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978 not found: ID does not exist" containerID="4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978" Oct 30 13:24:12.934732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.934590 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978"} err="failed to get container status \"4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978\": rpc error: code = NotFound desc = could not find container \"4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978\": container with ID starting with 4cb146a42b5a572f7fb861484926739efb5422a799cbb0b9d185e29b366be978 not found: ID does not exist" Oct 30 13:24:12.934732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.934652 5693 scope.go:122] "RemoveContainer" containerID="a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e" Oct 30 13:24:12.949006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.948986 5693 scope.go:122] "RemoveContainer" containerID="a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e" Oct 30 13:24:12.949334 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:12.949313 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e\": container with ID starting with a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e not found: ID does not exist" containerID="a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e" Oct 30 13:24:12.949414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.949341 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e"} err="failed to get container status \"a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e\": rpc error: code = NotFound desc = could not find container \"a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e\": container with ID starting with a22e53bd6cf37b93a24adc27fe8fa7ade83871575cbeb3ae77313792fcd9641e not found: ID does not exist" Oct 30 13:24:12.994934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.994903 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.994945 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995064 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995099 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995177 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995211 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.995271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995264 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995434 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.995477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995454 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995586 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.995675 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert\") pod \"5163624d-38bf-425b-bb38-dd6f1b14668a\" (UID: \"5163624d-38bf-425b-bb38-dd6f1b14668a\") " Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996246 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996479 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996503 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6gn\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-kube-api-access-9q6gn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996521 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5163624d-38bf-425b-bb38-dd6f1b14668a-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996537 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996551 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996565 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996579 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996593 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996624 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996638 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996652 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996669 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c3603-f114-4fd7-8201-263f8d428598-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996684 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996700 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996713 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996727 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996740 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c3603-f114-4fd7-8201-263f8d428598-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996754 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0a847-3d24-4406-b285-026bf3db0c77-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996779 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996793 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996808 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996820 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996840 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996854 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996867 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bqzm\" (UniqueName: \"kubernetes.io/projected/055c3603-f114-4fd7-8201-263f8d428598-kube-api-access-2bqzm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996881 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996897 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6v22\" (UniqueName: \"kubernetes.io/projected/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-kube-api-access-d6v22\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996911 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0a847-3d24-4406-b285-026bf3db0c77-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996927 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.996940 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c3603-f114-4fd7-8201-263f8d428598-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.997180 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.997466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.997283 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.998816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.997933 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:12.998816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.998046 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "kube-api-access-5zds7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:12.998816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.998062 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:12.998816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:12.998107 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token" pod "5163624d-38bf-425b-bb38-dd6f1b14668a" (UID: "5163624d-38bf-425b-bb38-dd6f1b14668a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:13.098465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.098426 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:13.098465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.098462 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zds7\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-kube-api-access-5zds7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:13.098934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.098478 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:13.098934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.098491 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:13.098934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.098504 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:13.098934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.098516 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5163624d-38bf-425b-bb38-dd6f1b14668a-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:13.226409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.226378 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc"] Oct 30 13:24:13.229563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.229539 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tzgbc"] Oct 30 13:24:13.278579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.278480 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh"] Oct 30 13:24:13.278768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.278585 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m82gh"] Oct 30 13:24:13.302699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.302666 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9"] Oct 30 13:24:13.306501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.306474 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-t9rp9"] Oct 30 13:24:13.599071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.598925 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:13.899733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.899644 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77" Oct 30 13:24:13.952961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.952931 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77"] Oct 30 13:24:13.956145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:13.956121 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-crb77"] Oct 30 13:24:14.256926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:14.256890 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055c3603-f114-4fd7-8201-263f8d428598" path="/var/lib/kubelet/pods/055c3603-f114-4fd7-8201-263f8d428598/volumes" Oct 30 13:24:14.257647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:14.257626 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" path="/var/lib/kubelet/pods/0f2d53dd-aa16-4bc1-8208-5fbdd164ed93/volumes" Oct 30 13:24:14.258259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:14.258241 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" path="/var/lib/kubelet/pods/5163624d-38bf-425b-bb38-dd6f1b14668a/volumes" Oct 30 13:24:14.259272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:14.259255 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" path="/var/lib/kubelet/pods/d8a0a847-3d24-4406-b285-026bf3db0c77/volumes" Oct 30 13:24:14.758807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:14.758775 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:24:14.758956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:14.758850 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:24:15.418908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:15.417828 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:17.071630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:17.060289 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.312709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.312673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.531706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.526169 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.709671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.708659 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf"] Oct 30 13:24:18.709851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709802 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" Oct 30 13:24:18.709851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709835 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" Oct 30 13:24:18.709973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709890 5693 cpu_manager.go:414] "containerMap: removing container" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" Oct 30 13:24:18.709973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709898 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" Oct 30 13:24:18.709973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709919 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" Oct 30 13:24:18.709973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709927 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" Oct 30 13:24:18.709973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709964 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" Oct 30 13:24:18.709973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.709972 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" Oct 30 13:24:18.710266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.710130 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2d53dd-aa16-4bc1-8208-5fbdd164ed93" containerName="istio-proxy" Oct 30 13:24:18.710266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.710146 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="055c3603-f114-4fd7-8201-263f8d428598" containerName="istio-proxy" Oct 30 13:24:18.710266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.710157 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a0a847-3d24-4406-b285-026bf3db0c77" containerName="istio-proxy" Oct 30 13:24:18.710266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.710173 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5163624d-38bf-425b-bb38-dd6f1b14668a" containerName="istio-proxy" Oct 30 13:24:18.830658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.829102 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:18.838297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.838265 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf"] Oct 30 13:24:18.881208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.881173 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3a82db-dockercfg-ql6q8\"" Oct 30 13:24:18.881399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.881289 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3a82db\"" Oct 30 13:24:18.890367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.890334 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh"] Oct 30 13:24:18.905268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.905230 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv96g\" (UniqueName: \"kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:18.905546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.905523 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:18.909716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.909498 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh"] Oct 30 13:24:18.911322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:18.911298 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.007063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.007029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv96g\" (UniqueName: \"kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:19.007256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.007092 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:19.007256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.007164 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.007256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.007240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrd5\" (UniqueName: \"kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5\") pod \"payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.007946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.007921 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:19.026502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.026478 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv96g\" (UniqueName: \"kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:19.108641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.108586 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.108812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.108743 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrd5\" (UniqueName: \"kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5\") pod \"payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.109297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.109278 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.127672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.127648 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrd5\" (UniqueName: \"kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5\") pod \"payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.210564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.210529 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:19.230866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.230799 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.450358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.450323 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:19.606054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.595291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:19.606054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.598958 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf"] Oct 30 13:24:19.635784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.634123 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh"] Oct 30 13:24:19.643072 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:24:19.643046 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcada2b38_3329_45b4_9759_aa7602a30c33.slice/crio-6c3f46369d94a483f6a0b8f174be0a32a87cdd455aaa8c3f095b6098ac7b57df WatchSource:0}: Error finding container 6c3f46369d94a483f6a0b8f174be0a32a87cdd455aaa8c3f095b6098ac7b57df: Status 404 returned error can't find the container with id 6c3f46369d94a483f6a0b8f174be0a32a87cdd455aaa8c3f095b6098ac7b57df Oct 30 13:24:19.731700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.731673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:19.956160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.956067 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" event={"ID":"cada2b38-3329-45b4-9759-aa7602a30c33","Type":"ContainerStarted","Data":"d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73"} Oct 30 13:24:19.956451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.956432 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" event={"ID":"cada2b38-3329-45b4-9759-aa7602a30c33","Type":"ContainerStarted","Data":"6c3f46369d94a483f6a0b8f174be0a32a87cdd455aaa8c3f095b6098ac7b57df"} Oct 30 13:24:19.956589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.956576 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:19.958014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.957983 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" event={"ID":"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96","Type":"ContainerStarted","Data":"7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01"} Oct 30 13:24:19.958283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.958261 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" event={"ID":"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96","Type":"ContainerStarted","Data":"aded8a2e4f6aab411c7e6ab47bb75342c3c6fa9b3d510f7f64379da033f29214"} Oct 30 13:24:19.958368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.958304 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:19.995097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:19.995045 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" podStartSLOduration=1.995030007 podStartE2EDuration="1.995030007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:19.994212549 +0000 UTC m=+1235.860830723" watchObservedRunningTime="2026-10-30 13:24:19.995030007 +0000 UTC m=+1235.861648183" Oct 30 13:24:20.032631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:20.032573 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" podStartSLOduration=2.032558693 podStartE2EDuration="2.032558693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:20.031497046 +0000 UTC m=+1235.898115223" watchObservedRunningTime="2026-10-30 13:24:20.032558693 +0000 UTC m=+1235.899176870" Oct 30 13:24:21.299415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:21.299375 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:23.083623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:23.080580 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:23.132558 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:23.132505 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:24:25.152897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:25.152860 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:25.298964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:25.298915 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:26.906540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:26.906450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:28.473123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:28.473081 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:30.128292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:30.128233 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:30.968807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:30.968765 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:24:30.969135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:30.969116 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:24:31.785625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:31.782250 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:32.754767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:32.754733 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:33.807442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:33.807406 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:36.369750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:36.369717 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:38.594155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:38.594101 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:40.292635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:40.292581 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:40.445762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:40.445720 5693 patch_prober.go:29] interesting pod/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:24:40.445952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:40.445798 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:24:42.022467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.022242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:42.142498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.142462 5693 generic.go:356] "Generic (PLEG): container finished" podID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerID="884e6297f64f7ccd8626b47b74b313623747fbd44341d64ecf1b45f3b043bbb3" exitCode=0 Oct 30 13:24:42.142684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.142535 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" event={"ID":"c466e2e1-ee81-4508-ae95-1c0e56fdef9a","Type":"ContainerDied","Data":"884e6297f64f7ccd8626b47b74b313623747fbd44341d64ecf1b45f3b043bbb3"} Oct 30 13:24:42.143042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.143024 5693 scope.go:122] "RemoveContainer" containerID="884e6297f64f7ccd8626b47b74b313623747fbd44341d64ecf1b45f3b043bbb3" Oct 30 13:24:42.144723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.144694 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:24:42.144808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.144761 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:24:42.144808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:42.144794 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:24:43.159641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:43.159564 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" event={"ID":"c466e2e1-ee81-4508-ae95-1c0e56fdef9a","Type":"ContainerStarted","Data":"38043d289ecfab34058b20967cb67e150501fe6fe1162a13c3ff8ded1a70d871"} Oct 30 13:24:43.744623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:43.739976 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:43.919612 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:43.919568 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:24:43.919781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:43.919736 5693 patch_prober.go:29] interesting pod/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:24:43.919825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:43.919788 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:24:44.919784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:44.919750 5693 patch_prober.go:29] interesting pod/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:24:44.920213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:44.919807 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:24:45.917990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:45.917953 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:45.920097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:45.920067 5693 patch_prober.go:29] interesting pod/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:24:45.920562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:45.920528 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:24:46.919744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:46.919718 5693 patch_prober.go:29] interesting pod/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:24:46.919820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:46.919780 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:24:47.309024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:47.308987 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc"] Oct 30 13:24:47.309806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:47.309770 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" podUID="4896d66c-cb07-443e-98c4-07282f59f232" containerName="maas-api" containerID="cri-o://9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b" gracePeriod=30 Oct 30 13:24:47.453165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:47.453140 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:47.934629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:47.931800 5693 patch_prober.go:29] interesting pod/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:24:47.934629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:47.931857 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:24:47.946629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:47.946021 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:24:48.042674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.039608 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca" pod "4896d66c-cb07-443e-98c4-07282f59f232" (UID: "4896d66c-cb07-443e-98c4-07282f59f232"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:48.042674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.039181 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca\") pod \"4896d66c-cb07-443e-98c4-07282f59f232\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " Oct 30 13:24:48.042674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.039832 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls\") pod \"4896d66c-cb07-443e-98c4-07282f59f232\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " Oct 30 13:24:48.042674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.040363 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs\") pod \"4896d66c-cb07-443e-98c4-07282f59f232\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " Oct 30 13:24:48.055723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.050505 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls" pod "4896d66c-cb07-443e-98c4-07282f59f232" (UID: "4896d66c-cb07-443e-98c4-07282f59f232"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:48.055723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.050778 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8\" (UniqueName: \"kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8\") pod \"4896d66c-cb07-443e-98c4-07282f59f232\" (UID: \"4896d66c-cb07-443e-98c4-07282f59f232\") " Oct 30 13:24:48.068763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.059441 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896d66c-cb07-443e-98c4-07282f59f232-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.068763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.059477 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.080918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.080861 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs" pod "4896d66c-cb07-443e-98c4-07282f59f232" (UID: "4896d66c-cb07-443e-98c4-07282f59f232"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:48.120282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.120212 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8" pod "4896d66c-cb07-443e-98c4-07282f59f232" (UID: "4896d66c-cb07-443e-98c4-07282f59f232"). InnerVolumeSpecName "kube-api-access-v8pd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:48.177623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.172230 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pd8\" (UniqueName: \"kubernetes.io/projected/4896d66c-cb07-443e-98c4-07282f59f232-kube-api-access-v8pd8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.177623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.172268 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896d66c-cb07-443e-98c4-07282f59f232-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.224006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.223302 5693 generic.go:356] "Generic (PLEG): container finished" podID="4896d66c-cb07-443e-98c4-07282f59f232" containerID="9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b" exitCode=0 Oct 30 13:24:48.224006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.223446 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" Oct 30 13:24:48.224006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.223695 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" event={"ID":"4896d66c-cb07-443e-98c4-07282f59f232","Type":"ContainerDied","Data":"9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b"} Oct 30 13:24:48.224006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.223756 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc" event={"ID":"4896d66c-cb07-443e-98c4-07282f59f232","Type":"ContainerDied","Data":"e814910b5dfb40276f0569e1d3eb0ad3c391286b9cfc1b1080baa58231eb9acd"} Oct 30 13:24:48.224006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.223785 5693 scope.go:122] "RemoveContainer" containerID="9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b" Oct 30 13:24:48.269795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.269773 5693 scope.go:122] "RemoveContainer" containerID="9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b" Oct 30 13:24:48.270201 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:48.270172 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b\": container with ID starting with 9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b not found: ID does not exist" containerID="9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b" Oct 30 13:24:48.270311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.270213 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b"} err="failed to get container status \"9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b\": rpc error: code = NotFound desc = could not find container \"9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b\": container with ID starting with 9489285101c8c216a100ee2e45ac084373886694c988df6f5a5d6902a30f905b not found: ID does not exist" Oct 30 13:24:48.290396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.290364 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc"] Oct 30 13:24:48.301520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.301491 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-121d75-5c677c9d5-4dlkc"] Oct 30 13:24:48.312501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.312464 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:48.460633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.460589 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:48.599079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.598993 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:48.734212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.734174 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:48.924522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.924444 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:24:48.924892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.924873 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:24:48.925529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:48.925511 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:24:49.000345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:49.000316 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:50.116085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.116051 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc"] Oct 30 13:24:50.116527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.116322 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" podUID="a8a88d50-bfee-45d5-a1b2-0d7dedde5853" containerName="maas-api" containerID="cri-o://38515da18833456b09557a463c7943f8e61e22fcddfe030366eda28280df53de" gracePeriod=30 Oct 30 13:24:50.222098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.222064 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:50.242374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.242341 5693 generic.go:356] "Generic (PLEG): container finished" podID="a8a88d50-bfee-45d5-a1b2-0d7dedde5853" containerID="38515da18833456b09557a463c7943f8e61e22fcddfe030366eda28280df53de" exitCode=0 Oct 30 13:24:50.242500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.242407 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" event={"ID":"a8a88d50-bfee-45d5-a1b2-0d7dedde5853","Type":"ContainerDied","Data":"38515da18833456b09557a463c7943f8e61e22fcddfe030366eda28280df53de"} Oct 30 13:24:50.261212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.261177 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4896d66c-cb07-443e-98c4-07282f59f232" path="/var/lib/kubelet/pods/4896d66c-cb07-443e-98c4-07282f59f232/volumes" Oct 30 13:24:50.402523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.402494 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:24:50.445481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.445448 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\") pod \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " Oct 30 13:24:50.446119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.446088 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs\") pod \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " Oct 30 13:24:50.446318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.446293 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x\" (UniqueName: \"kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x\") pod \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " Oct 30 13:24:50.446480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.446460 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca\") pod \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\" (UID: \"a8a88d50-bfee-45d5-a1b2-0d7dedde5853\") " Oct 30 13:24:50.447763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.447736 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca" pod "a8a88d50-bfee-45d5-a1b2-0d7dedde5853" (UID: "a8a88d50-bfee-45d5-a1b2-0d7dedde5853"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:50.448985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.448963 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls" pod "a8a88d50-bfee-45d5-a1b2-0d7dedde5853" (UID: "a8a88d50-bfee-45d5-a1b2-0d7dedde5853"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:50.459810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.459780 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x" pod "a8a88d50-bfee-45d5-a1b2-0d7dedde5853" (UID: "a8a88d50-bfee-45d5-a1b2-0d7dedde5853"). InnerVolumeSpecName "kube-api-access-6cc5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:50.460750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.460721 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs" pod "a8a88d50-bfee-45d5-a1b2-0d7dedde5853" (UID: "a8a88d50-bfee-45d5-a1b2-0d7dedde5853"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:50.472164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.472124 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:50.548411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.548372 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:50.548411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.548405 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:50.548635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.548418 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cc5x\" (UniqueName: \"kubernetes.io/projected/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-kube-api-access-6cc5x\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:50.548635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.548432 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a88d50-bfee-45d5-a1b2-0d7dedde5853-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:24:50.613693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.613662 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:50.919135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:50.919098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:51.065651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.065590 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:51.200496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.200403 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:51.251180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.251154 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" Oct 30 13:24:51.251344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.251196 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc" event={"ID":"a8a88d50-bfee-45d5-a1b2-0d7dedde5853","Type":"ContainerDied","Data":"8fdf007e28c07255a3e2f5ca02ed845f472c121d5df2addd69271d5deed2be62"} Oct 30 13:24:51.251344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.251251 5693 scope.go:122] "RemoveContainer" containerID="38515da18833456b09557a463c7943f8e61e22fcddfe030366eda28280df53de" Oct 30 13:24:51.301755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.301726 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc"] Oct 30 13:24:51.313689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.313663 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dcf84-847bd46cdb-mxsnc"] Oct 30 13:24:51.585591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.585551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:51.974690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:51.966194 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:52.257352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:52.257261 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a88d50-bfee-45d5-a1b2-0d7dedde5853" path="/var/lib/kubelet/pods/a8a88d50-bfee-45d5-a1b2-0d7dedde5853/volumes" Oct 30 13:24:52.910918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:52.910881 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:53.056622 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:24:53.056582 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:24:53.442755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:53.442723 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:54.723433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:54.723399 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:55.823831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:55.823796 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:55.968365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:55.968335 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:56.174334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:56.174245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:56.404318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:56.403108 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:56.607173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:56.607144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:57.641361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:57.641328 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:58.032576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.032543 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz"] Oct 30 13:24:58.033730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.033708 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a8a88d50-bfee-45d5-a1b2-0d7dedde5853" containerName="maas-api" Oct 30 13:24:58.033794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.033734 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a88d50-bfee-45d5-a1b2-0d7dedde5853" containerName="maas-api" Oct 30 13:24:58.033836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.033828 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4896d66c-cb07-443e-98c4-07282f59f232" containerName="maas-api" Oct 30 13:24:58.033872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.033837 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4896d66c-cb07-443e-98c4-07282f59f232" containerName="maas-api" Oct 30 13:24:58.034019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.034005 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4896d66c-cb07-443e-98c4-07282f59f232" containerName="maas-api" Oct 30 13:24:58.034055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.034025 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a88d50-bfee-45d5-a1b2-0d7dedde5853" containerName="maas-api" Oct 30 13:24:58.039843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.039816 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.056661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.056445 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz"] Oct 30 13:24:58.062402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.062379 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4091a5-openshift-default-dockercfg-djbhd\"" Oct 30 13:24:58.117102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117068 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117242 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117294 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117365 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117466 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117630 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117679 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.117855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.117758 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcgj\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.126797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.126774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222514 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222614 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222687 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222729 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222867 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222907 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222955 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.222980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcgj\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.223439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.223066 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.227433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.224302 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.227433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.224559 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.227433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.226171 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.240253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.240208 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.240529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.240507 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.248625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.244923 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.303532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.303439 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.366413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.363502 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.368547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.368419 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcgj\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj\") pod \"e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.572793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.572718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:58.663943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.663901 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:24:58.860177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.848697 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:58.985617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:58.985437 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q"] Oct 30 13:24:59.145297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.139426 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" containerID="cri-o://7e8ef3a94e4b36ff179b0449bd72385d9025dad5c87ce572b10a63935e1a11e0" gracePeriod=30 Oct 30 13:24:59.170240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.158483 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:59.174297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.173427 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:24:59.174297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.173525 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:24:59.174297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.173570 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:24:59.270819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.265041 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz"] Oct 30 13:24:59.405754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.405658 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" event={"ID":"accafbd8-9c8c-4493-8954-d5a5813391e0","Type":"ContainerStarted","Data":"70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7"} Oct 30 13:24:59.405754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.405724 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" event={"ID":"accafbd8-9c8c-4493-8954-d5a5813391e0","Type":"ContainerStarted","Data":"983bf65a556b506d7ecdfa256ecd30929b4924e2e3c3d9e5903d7bb95d4edcd5"} Oct 30 13:24:59.471216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.471144 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podStartSLOduration=1.471124493 podStartE2EDuration="1.471124493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:59.468609094 +0000 UTC m=+1275.335227259" watchObservedRunningTime="2026-10-30 13:24:59.471124493 +0000 UTC m=+1275.337742668" Oct 30 13:24:59.668385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:24:59.664831 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:00.314659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:00.314592 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:00.670361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:00.670279 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:01.426960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:01.426934 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:01.427938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:01.427920 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:01.881047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:01.881017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:02.045098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:02.045056 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:02.777180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:02.777083 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:03.495651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.495569 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:03.577861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.577802 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw"] Oct 30 13:25:03.583722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.583683 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.597926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.597894 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw"] Oct 30 13:25:03.605020 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.604996 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4091a5-dockercfg-ghwgp\"" Oct 30 13:25:03.650414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.650379 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpdr\" (UniqueName: \"kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr\") pod \"payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.650575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.650487 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.651265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.651245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:03.700619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.687583 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4091a5\"" Oct 30 13:25:03.817839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.817741 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpdr\" (UniqueName: \"kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr\") pod \"payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.818060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.817972 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.819010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.818982 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.867409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.866405 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpdr\" (UniqueName: \"kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr\") pod \"payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:03.907620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:03.904860 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:04.028625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.028534 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9"] Oct 30 13:25:04.053234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.053203 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln"] Oct 30 13:25:04.189539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.189446 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:04.364233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.364181 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw"] Oct 30 13:25:04.380188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.378749 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:04.460178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.459722 5693 generic.go:356] "Generic (PLEG): container finished" podID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerID="7e8ef3a94e4b36ff179b0449bd72385d9025dad5c87ce572b10a63935e1a11e0" exitCode=0 Oct 30 13:25:04.460178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.459932 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" event={"ID":"cad06448-7087-4e86-868c-6ccc0ecf709c","Type":"ContainerDied","Data":"7e8ef3a94e4b36ff179b0449bd72385d9025dad5c87ce572b10a63935e1a11e0"} Oct 30 13:25:04.460178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.460004 5693 scope.go:122] "RemoveContainer" containerID="8b280be808674f3220f3453da6a32ea46dd8fa7c71c312a1984d6a74d5d29b02" Oct 30 13:25:04.463347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.463172 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" event={"ID":"1e41bcb3-272d-486e-b0a8-26a713ce6e92","Type":"ContainerStarted","Data":"74d80f7adb20c75ee8720e73f1bab58ea579697129625c92561db959d489098a"} Oct 30 13:25:04.525901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.525876 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:25:04.551074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551046 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551088 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551176 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551248 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551293 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551372 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.551479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551422 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.551479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551443 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551503 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551544 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551578 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo\") pod \"cad06448-7087-4e86-868c-6ccc0ecf709c\" (UID: \"cad06448-7087-4e86-868c-6ccc0ecf709c\") " Oct 30 13:25:04.551835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551681 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.551891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551841 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.551891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.551852 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:04.552183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.552163 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.552183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.552184 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.552368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.552197 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.552368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.552209 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06448-7087-4e86-868c-6ccc0ecf709c-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.552368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.552220 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.553777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.553755 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:04.553777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.553764 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.554062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.554045 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:04.554108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.554087 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf" pod "cad06448-7087-4e86-868c-6ccc0ecf709c" (UID: "cad06448-7087-4e86-868c-6ccc0ecf709c"). InnerVolumeSpecName "kube-api-access-q5hbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:04.667507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.667455 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.723678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.711869 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.723678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.711901 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06448-7087-4e86-868c-6ccc0ecf709c-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.723678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:04.711915 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hbf\" (UniqueName: \"kubernetes.io/projected/cad06448-7087-4e86-868c-6ccc0ecf709c-kube-api-access-q5hbf\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:05.472918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.472881 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" event={"ID":"1e41bcb3-272d-486e-b0a8-26a713ce6e92","Type":"ContainerStarted","Data":"a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4"} Oct 30 13:25:05.473135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.472945 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:05.474312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.474290 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" Oct 30 13:25:05.474312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.474306 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q" event={"ID":"cad06448-7087-4e86-868c-6ccc0ecf709c","Type":"ContainerDied","Data":"043754f14e3c4f6db9b4fc71eee8ff3dfd93f6f34a06951372933dc5b110a555"} Oct 30 13:25:05.474506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.474351 5693 scope.go:122] "RemoveContainer" containerID="7e8ef3a94e4b36ff179b0449bd72385d9025dad5c87ce572b10a63935e1a11e0" Oct 30 13:25:05.511247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.511199 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" podStartSLOduration=2.511185686 podStartE2EDuration="2.511185686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:05.509023888 +0000 UTC m=+1281.375642087" watchObservedRunningTime="2026-10-30 13:25:05.511185686 +0000 UTC m=+1281.377803858" Oct 30 13:25:05.545936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.545901 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q"] Oct 30 13:25:05.549918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.549888 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dcf84-openshift-default-598558579-7dl2q"] Oct 30 13:25:05.603287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:05.603219 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:06.255844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:06.255811 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" path="/var/lib/kubelet/pods/cad06448-7087-4e86-868c-6ccc0ecf709c/volumes" Oct 30 13:25:06.606451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:06.606367 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-kg4wx_89c0aafb-9900-4eb3-90ec-d26f746bf51a/payload-processing/0.log" Oct 30 13:25:06.839258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:06.839233 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-kg4wx_89c0aafb-9900-4eb3-90ec-d26f746bf51a/payload-processing/0.log" Oct 30 13:25:07.025890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:07.025864 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk_439e921f-9e85-4bcd-a458-a7de27e0ca70/payload-processing/0.log" Oct 30 13:25:07.214502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:07.214470 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk_439e921f-9e85-4bcd-a458-a7de27e0ca70/payload-processing/0.log" Oct 30 13:25:07.302500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:07.302420 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.434755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:07.434718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:09.029903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.029841 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" podUID="61edd7bd-7327-4486-a019-867f2898c0ff" containerName="payload-processing" containerID="cri-o://87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330" gracePeriod=25 Oct 30 13:25:09.054202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.054166 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" podUID="454c01f1-0956-42b7-9ede-7791000dae14" containerName="payload-pre-processing" containerID="cri-o://e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796" gracePeriod=25 Oct 30 13:25:09.151098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.151066 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:09.415309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.415275 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:25:09.476154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.476114 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:25:09.505375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.505324 5693 generic.go:356] "Generic (PLEG): container finished" podID="454c01f1-0956-42b7-9ede-7791000dae14" containerID="e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796" exitCode=0 Oct 30 13:25:09.505550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.505384 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" Oct 30 13:25:09.505550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.505402 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" event={"ID":"454c01f1-0956-42b7-9ede-7791000dae14","Type":"ContainerDied","Data":"e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796"} Oct 30 13:25:09.505550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.505490 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln" event={"ID":"454c01f1-0956-42b7-9ede-7791000dae14","Type":"ContainerDied","Data":"5dc98ca46caf4afa423be4e944faafbd5b17a094ce5451bd867971b8e976ec00"} Oct 30 13:25:09.505550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.505515 5693 scope.go:122] "RemoveContainer" containerID="e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796" Oct 30 13:25:09.507389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.507357 5693 generic.go:356] "Generic (PLEG): container finished" podID="61edd7bd-7327-4486-a019-867f2898c0ff" containerID="87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330" exitCode=0 Oct 30 13:25:09.507531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.507404 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" Oct 30 13:25:09.507531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.507428 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" event={"ID":"61edd7bd-7327-4486-a019-867f2898c0ff","Type":"ContainerDied","Data":"87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330"} Oct 30 13:25:09.507531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.507461 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9" event={"ID":"61edd7bd-7327-4486-a019-867f2898c0ff","Type":"ContainerDied","Data":"4e95b022eb5d241cc3109a5840b0891a532c324feebe3c41d7ad10930ccce1db"} Oct 30 13:25:09.507767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.507742 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2\" (UniqueName: \"kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2\") pod \"454c01f1-0956-42b7-9ede-7791000dae14\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " Oct 30 13:25:09.507966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.507947 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume\") pod \"61edd7bd-7327-4486-a019-867f2898c0ff\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " Oct 30 13:25:09.508214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.508168 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume\") pod \"454c01f1-0956-42b7-9ede-7791000dae14\" (UID: \"454c01f1-0956-42b7-9ede-7791000dae14\") " Oct 30 13:25:09.508334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.508245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg\") pod \"61edd7bd-7327-4486-a019-867f2898c0ff\" (UID: \"61edd7bd-7327-4486-a019-867f2898c0ff\") " Oct 30 13:25:09.508414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.508341 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume" pod "61edd7bd-7327-4486-a019-867f2898c0ff" (UID: "61edd7bd-7327-4486-a019-867f2898c0ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:09.508980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.508955 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume" pod "454c01f1-0956-42b7-9ede-7791000dae14" (UID: "454c01f1-0956-42b7-9ede-7791000dae14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:09.509520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.509495 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454c01f1-0956-42b7-9ede-7791000dae14-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:09.509663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.509524 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61edd7bd-7327-4486-a019-867f2898c0ff-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:09.510246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.510227 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2" pod "454c01f1-0956-42b7-9ede-7791000dae14" (UID: "454c01f1-0956-42b7-9ede-7791000dae14"). InnerVolumeSpecName "kube-api-access-zh2f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:09.510491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.510462 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg" pod "61edd7bd-7327-4486-a019-867f2898c0ff" (UID: "61edd7bd-7327-4486-a019-867f2898c0ff"). InnerVolumeSpecName "kube-api-access-fdhxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:09.519940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.519908 5693 scope.go:122] "RemoveContainer" containerID="e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796" Oct 30 13:25:09.520284 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:09.520256 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796\": container with ID starting with e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796 not found: ID does not exist" containerID="e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796" Oct 30 13:25:09.520383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.520293 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796"} err="failed to get container status \"e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796\": rpc error: code = NotFound desc = could not find container \"e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796\": container with ID starting with e52060634fe53cb3803828144b6454059c82f60e3faa3d9c1dde944ef7b37796 not found: ID does not exist" Oct 30 13:25:09.520383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.520312 5693 scope.go:122] "RemoveContainer" containerID="87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330" Oct 30 13:25:09.536672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.536642 5693 scope.go:122] "RemoveContainer" containerID="87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330" Oct 30 13:25:09.537239 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:09.537206 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330\": container with ID starting with 87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330 not found: ID does not exist" containerID="87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330" Oct 30 13:25:09.537325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.537252 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330"} err="failed to get container status \"87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330\": rpc error: code = NotFound desc = could not find container \"87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330\": container with ID starting with 87c341dfa378bb52e52215e757cd1c44e7aaf203ba4b78c05c74604cd6b21330 not found: ID does not exist" Oct 30 13:25:09.611665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.611569 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/61edd7bd-7327-4486-a019-867f2898c0ff-kube-api-access-fdhxg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:09.611665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.611623 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2f2\" (UniqueName: \"kubernetes.io/projected/454c01f1-0956-42b7-9ede-7791000dae14-kube-api-access-zh2f2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:09.874733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.874648 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln"] Oct 30 13:25:09.886863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.886832 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9e946-844c87f899-rmvln"] Oct 30 13:25:09.927940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.927911 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9"] Oct 30 13:25:09.932012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:09.931987 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9e946-7f8b8986cd-89qp9"] Oct 30 13:25:10.255514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:10.255479 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454c01f1-0956-42b7-9ede-7791000dae14" path="/var/lib/kubelet/pods/454c01f1-0956-42b7-9ede-7791000dae14/volumes" Oct 30 13:25:10.255967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:10.255948 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61edd7bd-7327-4486-a019-867f2898c0ff" path="/var/lib/kubelet/pods/61edd7bd-7327-4486-a019-867f2898c0ff/volumes" Oct 30 13:25:11.479360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:11.476713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:13.072679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:13.072641 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:14.815251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:14.815184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:16.485308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:16.485273 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:25:16.550624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:16.542883 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:18.297670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:18.297632 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:19.879875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:19.879819 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:21.502787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:21.502747 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:22.153707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.153670 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r"] Oct 30 13:25:22.153925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.153900 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" containerID="cri-o://be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c" gracePeriod=30 Oct 30 13:25:22.156993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.156965 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw"] Oct 30 13:25:22.157187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.157167 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" containerID="cri-o://c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f" gracePeriod=30 Oct 30 13:25:22.159898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.159874 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg"] Oct 30 13:25:22.160125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.160079 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" containerID="cri-o://c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8" gracePeriod=30 Oct 30 13:25:22.163546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.163523 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z"] Oct 30 13:25:22.163698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:22.163681 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" containerID="cri-o://0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474" gracePeriod=30 Oct 30 13:25:23.059716 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:23.059676 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:25:23.543926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.543883 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q"] Oct 30 13:25:23.544665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544646 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" Oct 30 13:25:23.544665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544664 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" Oct 30 13:25:23.544808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544682 5693 cpu_manager.go:414] "containerMap: removing container" podUID="454c01f1-0956-42b7-9ede-7791000dae14" containerName="payload-pre-processing" Oct 30 13:25:23.544808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544688 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454c01f1-0956-42b7-9ede-7791000dae14" containerName="payload-pre-processing" Oct 30 13:25:23.544808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544721 5693 cpu_manager.go:414] "containerMap: removing container" podUID="61edd7bd-7327-4486-a019-867f2898c0ff" containerName="payload-processing" Oct 30 13:25:23.544808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544726 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61edd7bd-7327-4486-a019-867f2898c0ff" containerName="payload-processing" Oct 30 13:25:23.544931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544824 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="61edd7bd-7327-4486-a019-867f2898c0ff" containerName="payload-processing" Oct 30 13:25:23.544931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544837 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" Oct 30 13:25:23.544931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544844 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" Oct 30 13:25:23.544931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.544852 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="454c01f1-0956-42b7-9ede-7791000dae14" containerName="payload-pre-processing" Oct 30 13:25:23.549906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.549887 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.582639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.577632 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4091a5-serving-cert\"" Oct 30 13:25:23.583184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.583152 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q"] Oct 30 13:25:23.668202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.668154 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.668409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.668300 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.668507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.668487 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqpl\" (UniqueName: \"kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.668562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.668530 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.676352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.676324 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:23.781389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.781351 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqpl\" (UniqueName: \"kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.782349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.782315 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.789536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.789486 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.795419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.795295 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.798664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.783200 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.834449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.833013 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.835516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.835445 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:23.843286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.843253 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.848308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.848240 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqpl\" (UniqueName: \"kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl\") pod \"maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:23.868314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:23.868284 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:24.179959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:24.179933 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q"] Oct 30 13:25:24.182071 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:25:24.182038 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e17f6d6_8a57_4aa1_b52d_d00e374449e8.slice/crio-51c94cdf9bc5662e61d383e35551bb55cb5152b5e2faecf9f6deffe8fdb850fa WatchSource:0}: Error finding container 51c94cdf9bc5662e61d383e35551bb55cb5152b5e2faecf9f6deffe8fdb850fa: Status 404 returned error can't find the container with id 51c94cdf9bc5662e61d383e35551bb55cb5152b5e2faecf9f6deffe8fdb850fa Oct 30 13:25:24.183609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:24.183577 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:25:24.740556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:24.740518 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" event={"ID":"3e17f6d6-8a57-4aa1-b52d-d00e374449e8","Type":"ContainerStarted","Data":"51c94cdf9bc5662e61d383e35551bb55cb5152b5e2faecf9f6deffe8fdb850fa"} Oct 30 13:25:25.192916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:25.192883 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:25.752249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:25.752206 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" event={"ID":"3e17f6d6-8a57-4aa1-b52d-d00e374449e8","Type":"ContainerStarted","Data":"04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f"} Oct 30 13:25:25.753313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:25.753292 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:25.792477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:25.792426 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" podStartSLOduration=2.392947409 podStartE2EDuration="2.79240852s" totalImagesPullingTime="399.461111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:25.790433969 +0000 UTC m=+1301.657052142" watchObservedRunningTime="2026-10-30 13:25:25.79240852 +0000 UTC m=+1301.659026695" Oct 30 13:25:25.846827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:25.846796 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:26.000829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:26.000789 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:26.137638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:26.137525 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:26.554097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:26.554065 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:27.455792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.455763 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:25:27.520908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.520882 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:25:27.523757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.523731 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.523891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.523772 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.523891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.523800 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.523891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.523845 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.524054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.523924 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.524054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.523984 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.524054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.524029 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.524314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.524292 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.524967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.524731 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.524967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.524869 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw\") pod \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\" (UID: \"a735ba9b-70f3-4ea2-ae2d-22677e168c19\") " Oct 30 13:25:27.525266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.525245 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:25:27.525644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.525577 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.525763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.525704 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.525832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.525782 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.526299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.526174 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:27.526563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.526538 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.529953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.529928 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.531522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.531485 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "kube-api-access-mchhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.532972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.532938 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:27.535686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.535253 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:25:27.536251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.536228 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token" pod "a735ba9b-70f3-4ea2-ae2d-22677e168c19" (UID: "a735ba9b-70f3-4ea2-ae2d-22677e168c19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.627142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627100 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627153 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627199 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627241 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627270 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627305 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627351 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627401 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627423 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627483 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627510 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627553 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627650 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.627667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627664 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627725 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627765 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627855 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627887 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627933 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.627993 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data\") pod \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\" (UID: \"881e283c-6912-4130-8bef-7fa3f3ab9a8f\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628043 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628100 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628151 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628178 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628192 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628219 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628253 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628289 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.628356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628344 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy\") pod \"960c70b8-eefa-4af4-af60-6b509db15b39\" (UID: \"960c70b8-eefa-4af4-af60-6b509db15b39\") " Oct 30 13:25:27.629166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628641 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.629166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628644 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.629166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628843 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.629166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.628848 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.629408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.629383 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:27.629795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.629771 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.630311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.630200 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.630407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.630338 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.630811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.630679 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:27.630811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.630745 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.631054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.631028 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:27.631881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.631810 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.631966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.631932 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo\") pod \"1516a792-85d5-46b7-b2e3-230efeb5fe70\" (UID: \"1516a792-85d5-46b7-b2e3-230efeb5fe70\") " Oct 30 13:25:27.632870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632846 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.632967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632876 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.632967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632891 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.632967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632904 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.632967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632930 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c70b8-eefa-4af4-af60-6b509db15b39-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.632967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632944 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.632967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632958 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mchhw\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-kube-api-access-mchhw\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632972 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.632997 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633012 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633025 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633038 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633052 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633076 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633089 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633102 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633113 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633122 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a735ba9b-70f3-4ea2-ae2d-22677e168c19-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633129 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a735ba9b-70f3-4ea2-ae2d-22677e168c19-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633149 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633161 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1516a792-85d5-46b7-b2e3-230efeb5fe70-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633171 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.633334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.633180 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.635475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.635452 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.636396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.636361 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.637069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637019 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:27.637069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637024 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:27.637232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637154 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.637301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637268 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "kube-api-access-xtczw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.637373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637353 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:27.637466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637447 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "kube-api-access-jbrnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.637564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637546 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.637780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637758 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token" pod "881e283c-6912-4130-8bef-7fa3f3ab9a8f" (UID: "881e283c-6912-4130-8bef-7fa3f3ab9a8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.637868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.637819 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6" pod "960c70b8-eefa-4af4-af60-6b509db15b39" (UID: "960c70b8-eefa-4af4-af60-6b509db15b39"). InnerVolumeSpecName "kube-api-access-n47d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:27.639358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.639337 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo" pod "1516a792-85d5-46b7-b2e3-230efeb5fe70" (UID: "1516a792-85d5-46b7-b2e3-230efeb5fe70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:27.734889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734861 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734890 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtczw\" (UniqueName: \"kubernetes.io/projected/881e283c-6912-4130-8bef-7fa3f3ab9a8f-kube-api-access-xtczw\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734902 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734913 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734921 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881e283c-6912-4130-8bef-7fa3f3ab9a8f-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734928 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47d6\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-kube-api-access-n47d6\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734936 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734945 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrnp\" (UniqueName: \"kubernetes.io/projected/1516a792-85d5-46b7-b2e3-230efeb5fe70-kube-api-access-jbrnp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734952 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c70b8-eefa-4af4-af60-6b509db15b39-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734960 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c70b8-eefa-4af4-af60-6b509db15b39-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734967 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c70b8-eefa-4af4-af60-6b509db15b39-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.735050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.734976 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1516a792-85d5-46b7-b2e3-230efeb5fe70-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:27.774181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.774140 5693 generic.go:356] "Generic (PLEG): container finished" podID="960c70b8-eefa-4af4-af60-6b509db15b39" containerID="be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c" exitCode=0 Oct 30 13:25:27.774379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.774195 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" Oct 30 13:25:27.774379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.774230 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" event={"ID":"960c70b8-eefa-4af4-af60-6b509db15b39","Type":"ContainerDied","Data":"be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c"} Oct 30 13:25:27.774379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.774329 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r" event={"ID":"960c70b8-eefa-4af4-af60-6b509db15b39","Type":"ContainerDied","Data":"e352ebafea352dfd78fcdfe0ee01a63f007e18771f5452575db48198638028e1"} Oct 30 13:25:27.774379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.774360 5693 scope.go:122] "RemoveContainer" containerID="be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c" Oct 30 13:25:27.775587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.775558 5693 generic.go:356] "Generic (PLEG): container finished" podID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerID="0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474" exitCode=0 Oct 30 13:25:27.775688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.775616 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" event={"ID":"1516a792-85d5-46b7-b2e3-230efeb5fe70","Type":"ContainerDied","Data":"0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474"} Oct 30 13:25:27.775688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.775644 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" Oct 30 13:25:27.775688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.775655 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z" event={"ID":"1516a792-85d5-46b7-b2e3-230efeb5fe70","Type":"ContainerDied","Data":"9fa7e03a4300dc059db31355a8eb206204e0c9a9abe056220b07b84618fd5dc7"} Oct 30 13:25:27.777371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.777346 5693 generic.go:356] "Generic (PLEG): container finished" podID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerID="c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8" exitCode=0 Oct 30 13:25:27.777535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.777386 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" Oct 30 13:25:27.777535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.777386 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" event={"ID":"a735ba9b-70f3-4ea2-ae2d-22677e168c19","Type":"ContainerDied","Data":"c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8"} Oct 30 13:25:27.777535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.777466 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg" event={"ID":"a735ba9b-70f3-4ea2-ae2d-22677e168c19","Type":"ContainerDied","Data":"89a3f1764813d18cf1f6340073fe4c13fada23d3538869fb6560af75395dc99c"} Oct 30 13:25:27.779551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.779524 5693 generic.go:356] "Generic (PLEG): container finished" podID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerID="c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f" exitCode=0 Oct 30 13:25:27.780642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.780579 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" event={"ID":"881e283c-6912-4130-8bef-7fa3f3ab9a8f","Type":"ContainerDied","Data":"c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f"} Oct 30 13:25:27.780743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.780661 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" event={"ID":"881e283c-6912-4130-8bef-7fa3f3ab9a8f","Type":"ContainerDied","Data":"a3238bfa9f5bfc5f7ceb24886b65a124175fc081a21d3d765d310b151c7e2954"} Oct 30 13:25:27.780743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.780735 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw" Oct 30 13:25:27.788560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.788540 5693 scope.go:122] "RemoveContainer" containerID="be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c" Oct 30 13:25:27.788957 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:27.788921 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c\": container with ID starting with be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c not found: ID does not exist" containerID="be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c" Oct 30 13:25:27.789029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.788961 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c"} err="failed to get container status \"be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c\": rpc error: code = NotFound desc = could not find container \"be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c\": container with ID starting with be6ab76dc83ff0285292da74cf3bd241256c05a1cebbd01629130f8f1b09f20c not found: ID does not exist" Oct 30 13:25:27.789029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.788985 5693 scope.go:122] "RemoveContainer" containerID="0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474" Oct 30 13:25:27.799212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.799196 5693 scope.go:122] "RemoveContainer" containerID="0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474" Oct 30 13:25:27.799469 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:27.799450 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474\": container with ID starting with 0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474 not found: ID does not exist" containerID="0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474" Oct 30 13:25:27.799521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.799476 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474"} err="failed to get container status \"0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474\": rpc error: code = NotFound desc = could not find container \"0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474\": container with ID starting with 0ac7a2347c3ff695e6a8ff381ba14d8830733ded5222a027dff2e53f5e63a474 not found: ID does not exist" Oct 30 13:25:27.799521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.799491 5693 scope.go:122] "RemoveContainer" containerID="c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8" Oct 30 13:25:27.809404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.809378 5693 scope.go:122] "RemoveContainer" containerID="c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8" Oct 30 13:25:27.809732 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:27.809713 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8\": container with ID starting with c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8 not found: ID does not exist" containerID="c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8" Oct 30 13:25:27.809823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.809737 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8"} err="failed to get container status \"c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8\": rpc error: code = NotFound desc = could not find container \"c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8\": container with ID starting with c830cb7f34231b7622f33e4eabf2dcf42721a2eb799b29d20b42329d399cebf8 not found: ID does not exist" Oct 30 13:25:27.809823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.809755 5693 scope.go:122] "RemoveContainer" containerID="c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f" Oct 30 13:25:27.818841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.818821 5693 scope.go:122] "RemoveContainer" containerID="c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f" Oct 30 13:25:27.819089 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:27.819070 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f\": container with ID starting with c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f not found: ID does not exist" containerID="c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f" Oct 30 13:25:27.819153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.819099 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f"} err="failed to get container status \"c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f\": rpc error: code = NotFound desc = could not find container \"c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f\": container with ID starting with c1e26063fbaf8e2704992cadf9c17c1ce0c158b49583d824557b45dd0df9941f not found: ID does not exist" Oct 30 13:25:27.868566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.868539 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw"] Oct 30 13:25:27.871566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.871545 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9k6mw"] Oct 30 13:25:27.894737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.894712 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:27.932047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.931995 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r"] Oct 30 13:25:27.935228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.935197 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-k7g9r"] Oct 30 13:25:27.986615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.986488 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z"] Oct 30 13:25:27.990893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:27.990852 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-8zs6z"] Oct 30 13:25:28.026664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:28.021011 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg"] Oct 30 13:25:28.049662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:28.042217 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-cjzwg"] Oct 30 13:25:28.254787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:28.254714 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" path="/var/lib/kubelet/pods/1516a792-85d5-46b7-b2e3-230efeb5fe70/volumes" Oct 30 13:25:28.255259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:28.255243 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" path="/var/lib/kubelet/pods/881e283c-6912-4130-8bef-7fa3f3ab9a8f/volumes" Oct 30 13:25:28.255748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:28.255733 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" path="/var/lib/kubelet/pods/960c70b8-eefa-4af4-af60-6b509db15b39/volumes" Oct 30 13:25:28.256533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:28.256518 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" path="/var/lib/kubelet/pods/a735ba9b-70f3-4ea2-ae2d-22677e168c19/volumes" Oct 30 13:25:29.166976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.166942 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq"] Oct 30 13:25:29.167475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.167269 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" podUID="4c59269e-5a35-4147-bab4-74aabaef0898" containerName="maas-api" containerID="cri-o://296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60" gracePeriod=30 Oct 30 13:25:29.290046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.290014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:29.318287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.318252 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh"] Oct 30 13:25:29.318489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.318468 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" podUID="cada2b38-3329-45b4-9759-aa7602a30c33" containerName="payload-processing" containerID="cri-o://d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73" gracePeriod=30 Oct 30 13:25:29.343551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.340010 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf"] Oct 30 13:25:29.343551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.340227 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" podUID="e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" containerName="payload-pre-processing" containerID="cri-o://7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01" gracePeriod=30 Oct 30 13:25:29.440076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.439951 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:29.512388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.512360 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:25:29.563075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.563042 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs\") pod \"4c59269e-5a35-4147-bab4-74aabaef0898\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " Oct 30 13:25:29.563240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.563194 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca\") pod \"4c59269e-5a35-4147-bab4-74aabaef0898\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " Oct 30 13:25:29.563389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.563319 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls\") pod \"4c59269e-5a35-4147-bab4-74aabaef0898\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " Oct 30 13:25:29.563489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.563450 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2\" (UniqueName: \"kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2\") pod \"4c59269e-5a35-4147-bab4-74aabaef0898\" (UID: \"4c59269e-5a35-4147-bab4-74aabaef0898\") " Oct 30 13:25:29.563489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.563746 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca" pod "4c59269e-5a35-4147-bab4-74aabaef0898" (UID: "4c59269e-5a35-4147-bab4-74aabaef0898"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:29.563489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.564092 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c59269e-5a35-4147-bab4-74aabaef0898-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.573088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.570771 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls" pod "4c59269e-5a35-4147-bab4-74aabaef0898" (UID: "4c59269e-5a35-4147-bab4-74aabaef0898"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:29.573088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.570806 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2" pod "4c59269e-5a35-4147-bab4-74aabaef0898" (UID: "4c59269e-5a35-4147-bab4-74aabaef0898"). InnerVolumeSpecName "kube-api-access-g8xv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:29.573088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.570817 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs" pod "4c59269e-5a35-4147-bab4-74aabaef0898" (UID: "4c59269e-5a35-4147-bab4-74aabaef0898"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:29.577224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.577193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:29.633559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.633232 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:25:29.664877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.664842 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume\") pod \"cada2b38-3329-45b4-9759-aa7602a30c33\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " Oct 30 13:25:29.665038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.664928 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5\" (UniqueName: \"kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5\") pod \"cada2b38-3329-45b4-9759-aa7602a30c33\" (UID: \"cada2b38-3329-45b4-9759-aa7602a30c33\") " Oct 30 13:25:29.665200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.665178 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume" pod "cada2b38-3329-45b4-9759-aa7602a30c33" (UID: "cada2b38-3329-45b4-9759-aa7602a30c33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:29.665576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.665552 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cada2b38-3329-45b4-9759-aa7602a30c33-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.665576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.665575 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xv2\" (UniqueName: \"kubernetes.io/projected/4c59269e-5a35-4147-bab4-74aabaef0898-kube-api-access-g8xv2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.665740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.665585 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.665740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.665611 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c59269e-5a35-4147-bab4-74aabaef0898-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.666952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.666936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5" pod "cada2b38-3329-45b4-9759-aa7602a30c33" (UID: "cada2b38-3329-45b4-9759-aa7602a30c33"). InnerVolumeSpecName "kube-api-access-fhrd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:29.781271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.774892 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrd5\" (UniqueName: \"kubernetes.io/projected/cada2b38-3329-45b4-9759-aa7602a30c33-kube-api-access-fhrd5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.840738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.840353 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:25:29.857214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.857179 5693 generic.go:356] "Generic (PLEG): container finished" podID="4c59269e-5a35-4147-bab4-74aabaef0898" containerID="296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60" exitCode=0 Oct 30 13:25:29.857552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.857529 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" Oct 30 13:25:29.860284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.857711 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" event={"ID":"4c59269e-5a35-4147-bab4-74aabaef0898","Type":"ContainerDied","Data":"296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60"} Oct 30 13:25:29.860390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.860362 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq" event={"ID":"4c59269e-5a35-4147-bab4-74aabaef0898","Type":"ContainerDied","Data":"c86e0b6ea62eba3ab482f4169147cc70f30b4c97c4ca1ff8dc12f6d15de45a1d"} Oct 30 13:25:29.860705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.860680 5693 scope.go:122] "RemoveContainer" containerID="296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60" Oct 30 13:25:29.861423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.861223 5693 generic.go:356] "Generic (PLEG): container finished" podID="cada2b38-3329-45b4-9759-aa7602a30c33" containerID="d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73" exitCode=0 Oct 30 13:25:29.861423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.861279 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" Oct 30 13:25:29.861423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.861333 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" event={"ID":"cada2b38-3329-45b4-9759-aa7602a30c33","Type":"ContainerDied","Data":"d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73"} Oct 30 13:25:29.861423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.861375 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh" event={"ID":"cada2b38-3329-45b4-9759-aa7602a30c33","Type":"ContainerDied","Data":"6c3f46369d94a483f6a0b8f174be0a32a87cdd455aaa8c3f095b6098ac7b57df"} Oct 30 13:25:29.868041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.867727 5693 generic.go:356] "Generic (PLEG): container finished" podID="e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" containerID="7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01" exitCode=0 Oct 30 13:25:29.868041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.867854 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" event={"ID":"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96","Type":"ContainerDied","Data":"7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01"} Oct 30 13:25:29.868041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.867882 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" event={"ID":"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96","Type":"ContainerDied","Data":"aded8a2e4f6aab411c7e6ab47bb75342c3c6fa9b3d510f7f64379da033f29214"} Oct 30 13:25:29.868041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.867927 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf" Oct 30 13:25:29.885690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.885663 5693 scope.go:122] "RemoveContainer" containerID="296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60" Oct 30 13:25:29.889123 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:29.888976 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60\": container with ID starting with 296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60 not found: ID does not exist" containerID="296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60" Oct 30 13:25:29.889123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.889020 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60"} err="failed to get container status \"296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60\": rpc error: code = NotFound desc = could not find container \"296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60\": container with ID starting with 296cfbda48df86dc8887fbdba412ad3e62a010050e50e92543d425367dc1eb60 not found: ID does not exist" Oct 30 13:25:29.889123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.889043 5693 scope.go:122] "RemoveContainer" containerID="d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73" Oct 30 13:25:29.906044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.906026 5693 scope.go:122] "RemoveContainer" containerID="d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73" Oct 30 13:25:29.906556 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:29.906524 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73\": container with ID starting with d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73 not found: ID does not exist" containerID="d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73" Oct 30 13:25:29.906709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.906564 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73"} err="failed to get container status \"d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73\": rpc error: code = NotFound desc = could not find container \"d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73\": container with ID starting with d4057b98a8d793d9cda6e43c26499fa56068c7a5233d863a04eee0e501b6bc73 not found: ID does not exist" Oct 30 13:25:29.906709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.906586 5693 scope.go:122] "RemoveContainer" containerID="7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01" Oct 30 13:25:29.913397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.913372 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:29.921420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.921401 5693 scope.go:122] "RemoveContainer" containerID="7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01" Oct 30 13:25:29.921753 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:29.921732 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01\": container with ID starting with 7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01 not found: ID does not exist" containerID="7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01" Oct 30 13:25:29.921851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.921763 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01"} err="failed to get container status \"7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01\": rpc error: code = NotFound desc = could not find container \"7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01\": container with ID starting with 7f8489d242241a04b3ab444cb6debe89b66127059e6265b9715a73da62cb8c01 not found: ID does not exist" Oct 30 13:25:29.930075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.930056 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh"] Oct 30 13:25:29.934905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.934874 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-69cc8b9ffb-fzqsh"] Oct 30 13:25:29.967229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.967171 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq"] Oct 30 13:25:29.973267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.973245 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3a82db-85bf79df4f-25vtq"] Oct 30 13:25:29.979251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.979226 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume\") pod \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " Oct 30 13:25:29.979343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.979268 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g\" (UniqueName: \"kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g\") pod \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\" (UID: \"e7da0d9b-0e3d-4f6c-a628-4584a98fbc96\") " Oct 30 13:25:29.979711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.979687 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume" pod "e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" (UID: "e7da0d9b-0e3d-4f6c-a628-4584a98fbc96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:29.979888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.979871 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:29.981387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:29.981364 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g" pod "e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" (UID: "e7da0d9b-0e3d-4f6c-a628-4584a98fbc96"). InnerVolumeSpecName "kube-api-access-mv96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:30.048520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.048487 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:30.082922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.082844 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv96g\" (UniqueName: \"kubernetes.io/projected/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96-kube-api-access-mv96g\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:30.185016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.184979 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:30.215561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.215533 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf"] Oct 30 13:25:30.219073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.219019 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-656fc5d695-fj4rf"] Oct 30 13:25:30.255281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.255253 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c59269e-5a35-4147-bab4-74aabaef0898" path="/var/lib/kubelet/pods/4c59269e-5a35-4147-bab4-74aabaef0898/volumes" Oct 30 13:25:30.255700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.255686 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cada2b38-3329-45b4-9759-aa7602a30c33" path="/var/lib/kubelet/pods/cada2b38-3329-45b4-9759-aa7602a30c33/volumes" Oct 30 13:25:30.256097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.256068 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" path="/var/lib/kubelet/pods/e7da0d9b-0e3d-4f6c-a628-4584a98fbc96/volumes" Oct 30 13:25:30.533453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:30.533417 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:31.855756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:31.854779 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:32.785927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:32.785899 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:25:33.409851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:33.409822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:33.551376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:33.551341 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:33.760918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:33.760872 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:35.399641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:35.399583 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:37.027720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:37.027674 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:37.330848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:37.330763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:37.466839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:37.466804 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:38.956358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:38.956318 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:39.357655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.355111 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:39.358332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.358295 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn"] Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359316 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359341 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad06448-7087-4e86-868c-6ccc0ecf709c" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359372 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4c59269e-5a35-4147-bab4-74aabaef0898" containerName="maas-api" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359388 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c59269e-5a35-4147-bab4-74aabaef0898" containerName="maas-api" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359400 5693 cpu_manager.go:414] "containerMap: removing container" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359408 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359426 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" containerName="payload-pre-processing" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359433 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" containerName="payload-pre-processing" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359457 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cada2b38-3329-45b4-9759-aa7602a30c33" containerName="payload-processing" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359464 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cada2b38-3329-45b4-9759-aa7602a30c33" containerName="payload-processing" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359490 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359496 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359524 5693 cpu_manager.go:414] "containerMap: removing container" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359531 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359551 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359559 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359726 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a735ba9b-70f3-4ea2-ae2d-22677e168c19" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359745 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="881e283c-6912-4130-8bef-7fa3f3ab9a8f" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359759 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1516a792-85d5-46b7-b2e3-230efeb5fe70" containerName="istio-proxy" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359775 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c59269e-5a35-4147-bab4-74aabaef0898" containerName="maas-api" Oct 30 13:25:39.359772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359786 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="960c70b8-eefa-4af4-af60-6b509db15b39" containerName="istio-proxy" Oct 30 13:25:39.360908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359797 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7da0d9b-0e3d-4f6c-a628-4584a98fbc96" containerName="payload-pre-processing" Oct 30 13:25:39.360908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.359808 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cada2b38-3329-45b4-9759-aa7602a30c33" containerName="payload-processing" Oct 30 13:25:39.364178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.364150 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.377018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.376972 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn"] Oct 30 13:25:39.383373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.383350 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d797fc-openshift-default-dockercfg-52l4g\"" Oct 30 13:25:39.423737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.423693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.423918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.423865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.423997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.423934 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.424056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.424022 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwdp\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.424111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.424079 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.424203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.424182 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.424247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.424217 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.424308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.424292 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.424395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.424380 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.538922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.538676 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.538922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.538798 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.538922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.538830 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwdp\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.538922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.538905 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.539342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.538965 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.539342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.538991 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.539342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.539027 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.539342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.539088 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.539342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.539190 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.540580 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.541743 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.542554 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.542831 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.543057 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.544131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.546764 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.553621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.547256 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.600623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.596678 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwdp\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.619680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.605534 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token\") pod \"e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.695022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.694959 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:39.906684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:39.881167 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:40.057217 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:25:40.057177 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc479868_520a_45f0_bc7c_ba06a8f95830.slice/crio-17ff5af89e65a0b8b7c7cfc6e5c2d5ff7fbe19d869670b3384a01203d98d2451 WatchSource:0}: Error finding container 17ff5af89e65a0b8b7c7cfc6e5c2d5ff7fbe19d869670b3384a01203d98d2451: Status 404 returned error can't find the container with id 17ff5af89e65a0b8b7c7cfc6e5c2d5ff7fbe19d869670b3384a01203d98d2451 Oct 30 13:25:40.058360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:40.058336 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn"] Oct 30 13:25:40.060497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:40.060460 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:25:40.060588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:40.060538 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:25:40.060588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:40.060580 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:25:40.981322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:40.981282 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" event={"ID":"cc479868-520a-45f0-bc7c-ba06a8f95830","Type":"ContainerStarted","Data":"ea60ef3bb25e9934a7526f1bec552506158275661eb2419b9d7a5f5c3b0698d5"} Oct 30 13:25:40.981553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:40.981358 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" event={"ID":"cc479868-520a-45f0-bc7c-ba06a8f95830","Type":"ContainerStarted","Data":"17ff5af89e65a0b8b7c7cfc6e5c2d5ff7fbe19d869670b3384a01203d98d2451"} Oct 30 13:25:41.056352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:41.056286 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podStartSLOduration=2.056265493 podStartE2EDuration="2.056265493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:41.049738832 +0000 UTC m=+1316.916357006" watchObservedRunningTime="2026-10-30 13:25:41.056265493 +0000 UTC m=+1316.922883666" Oct 30 13:25:41.495873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:41.495842 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:41.695224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:41.695185 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:41.701214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:41.701189 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:41.988435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:41.988402 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:41.989442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:41.989423 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:25:43.521716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:43.521680 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq"] Oct 30 13:25:43.566981 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:43.566938 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" podUID="841bf8bd-a836-4214-a788-17025ef8e75e" containerName="maas-api" containerID="cri-o://1880c6526a906744a04e587299e872ca2fd4c1bc2f9b58a92585d098345eeacb" gracePeriod=30 Oct 30 13:25:43.569256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:43.569145 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:43.708887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:43.708848 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:44.049837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.046787 5693 generic.go:356] "Generic (PLEG): container finished" podID="841bf8bd-a836-4214-a788-17025ef8e75e" containerID="1880c6526a906744a04e587299e872ca2fd4c1bc2f9b58a92585d098345eeacb" exitCode=0 Oct 30 13:25:44.053805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.053746 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" event={"ID":"841bf8bd-a836-4214-a788-17025ef8e75e","Type":"ContainerDied","Data":"1880c6526a906744a04e587299e872ca2fd4c1bc2f9b58a92585d098345eeacb"} Oct 30 13:25:44.139722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.139692 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:25:44.273554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.273342 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca\") pod \"841bf8bd-a836-4214-a788-17025ef8e75e\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " Oct 30 13:25:44.275856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.275784 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs\") pod \"841bf8bd-a836-4214-a788-17025ef8e75e\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " Oct 30 13:25:44.277398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.277376 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p\" (UniqueName: \"kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p\") pod \"841bf8bd-a836-4214-a788-17025ef8e75e\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " Oct 30 13:25:44.277617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.277588 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\") pod \"841bf8bd-a836-4214-a788-17025ef8e75e\" (UID: \"841bf8bd-a836-4214-a788-17025ef8e75e\") " Oct 30 13:25:44.280878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.280853 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca" pod "841bf8bd-a836-4214-a788-17025ef8e75e" (UID: "841bf8bd-a836-4214-a788-17025ef8e75e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:44.294019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.293973 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls" pod "841bf8bd-a836-4214-a788-17025ef8e75e" (UID: "841bf8bd-a836-4214-a788-17025ef8e75e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:44.294249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.294229 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs" pod "841bf8bd-a836-4214-a788-17025ef8e75e" (UID: "841bf8bd-a836-4214-a788-17025ef8e75e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:44.309630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.309528 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p" pod "841bf8bd-a836-4214-a788-17025ef8e75e" (UID: "841bf8bd-a836-4214-a788-17025ef8e75e"). InnerVolumeSpecName "kube-api-access-srb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:44.380281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.380249 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.380423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.380287 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/841bf8bd-a836-4214-a788-17025ef8e75e-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.380423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.380304 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841bf8bd-a836-4214-a788-17025ef8e75e-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.380423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.380318 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srb4p\" (UniqueName: \"kubernetes.io/projected/841bf8bd-a836-4214-a788-17025ef8e75e-kube-api-access-srb4p\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.613892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.613863 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:44.759271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:44.759188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:45.056296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.056240 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:45.057732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.057710 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" Oct 30 13:25:45.057888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.057747 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq" event={"ID":"841bf8bd-a836-4214-a788-17025ef8e75e","Type":"ContainerDied","Data":"2b99f416b1943a437af1ee3881b80276cf8fac9a822a9222f6fff2abc2a55294"} Oct 30 13:25:45.057888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.057805 5693 scope.go:122] "RemoveContainer" containerID="1880c6526a906744a04e587299e872ca2fd4c1bc2f9b58a92585d098345eeacb" Oct 30 13:25:45.112453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.112406 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq"] Oct 30 13:25:45.142635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.142555 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fb1a8-6c76d94498-hgrfq"] Oct 30 13:25:45.214018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.213977 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:45.381081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:45.380984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.212708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.212650 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.255488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.255456 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841bf8bd-a836-4214-a788-17025ef8e75e" path="/var/lib/kubelet/pods/841bf8bd-a836-4214-a788-17025ef8e75e/volumes" Oct 30 13:25:46.287363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.287326 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc"] Oct 30 13:25:46.288086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.288033 5693 cpu_manager.go:414] "containerMap: removing container" podUID="841bf8bd-a836-4214-a788-17025ef8e75e" containerName="maas-api" Oct 30 13:25:46.288086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.288050 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841bf8bd-a836-4214-a788-17025ef8e75e" containerName="maas-api" Oct 30 13:25:46.288279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.288244 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="841bf8bd-a836-4214-a788-17025ef8e75e" containerName="maas-api" Oct 30 13:25:46.293246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.293224 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.310323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.310291 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc"] Oct 30 13:25:46.326976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.326944 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d797fc\"" Oct 30 13:25:46.332112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.329089 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d797fc-dockercfg-78nrq\"" Oct 30 13:25:46.335405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.335109 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2"] Oct 30 13:25:46.335405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.335333 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" podUID="1285ceff-bd1a-4d05-9528-63b3328b6002" containerName="maas-api" containerID="cri-o://d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870" gracePeriod=30 Oct 30 13:25:46.350018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.340927 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67"] Oct 30 13:25:46.350018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.346821 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.378829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.370821 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67"] Oct 30 13:25:46.378829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.372338 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.424227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.424187 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.424410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.424332 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.424410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.424388 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xg9\" (UniqueName: \"kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9\") pod \"payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.424538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.424513 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ds7\" (UniqueName: \"kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7\") pod \"payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.510650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.507389 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz"] Oct 30 13:25:46.510650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.507915 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" podUID="cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" containerName="payload-pre-processing" containerID="cri-o://016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9" gracePeriod=30 Oct 30 13:25:46.555795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.548127 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ds7\" (UniqueName: \"kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7\") pod \"payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.555795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.548213 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.555795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.548320 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.555795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.548364 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xg9\" (UniqueName: \"kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9\") pod \"payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.555795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.549570 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.555795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.550366 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.558885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.558852 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.570036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.569994 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xg9\" (UniqueName: \"kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9\") pod \"payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.598571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.598540 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ds7\" (UniqueName: \"kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7\") pod \"payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.618400 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.618370 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:46.686650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.686148 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:25:46.698055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.698026 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.703635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.703508 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:46.752281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.752246 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls\") pod \"1285ceff-bd1a-4d05-9528-63b3328b6002\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " Oct 30 13:25:46.752383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.752324 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca\") pod \"1285ceff-bd1a-4d05-9528-63b3328b6002\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " Oct 30 13:25:46.752383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.752364 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc\" (UniqueName: \"kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc\") pod \"1285ceff-bd1a-4d05-9528-63b3328b6002\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " Oct 30 13:25:46.752514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.752496 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs\") pod \"1285ceff-bd1a-4d05-9528-63b3328b6002\" (UID: \"1285ceff-bd1a-4d05-9528-63b3328b6002\") " Oct 30 13:25:46.755097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.754366 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca" pod "1285ceff-bd1a-4d05-9528-63b3328b6002" (UID: "1285ceff-bd1a-4d05-9528-63b3328b6002"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:46.765928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.765880 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc" pod "1285ceff-bd1a-4d05-9528-63b3328b6002" (UID: "1285ceff-bd1a-4d05-9528-63b3328b6002"). InnerVolumeSpecName "kube-api-access-72hlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:46.768226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.768181 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls" pod "1285ceff-bd1a-4d05-9528-63b3328b6002" (UID: "1285ceff-bd1a-4d05-9528-63b3328b6002"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:46.772490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.772465 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs" pod "1285ceff-bd1a-4d05-9528-63b3328b6002" (UID: "1285ceff-bd1a-4d05-9528-63b3328b6002"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:46.890190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.855288 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.890190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.855322 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1285ceff-bd1a-4d05-9528-63b3328b6002-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.890190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.855337 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hlc\" (UniqueName: \"kubernetes.io/projected/1285ceff-bd1a-4d05-9528-63b3328b6002-kube-api-access-72hlc\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.890190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:46.855350 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1285ceff-bd1a-4d05-9528-63b3328b6002-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:47.093305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.091289 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc"] Oct 30 13:25:47.097633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.097335 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:25:47.118738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.118154 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" event={"ID":"cc06a9a5-c433-4023-95e7-861839c2266f","Type":"ContainerStarted","Data":"67f1e70af0c7072265b92317b198f6e9723d0cf3a4aac56f2fd4df98ee9af82d"} Oct 30 13:25:47.121234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.120279 5693 generic.go:356] "Generic (PLEG): container finished" podID="cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" containerID="016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9" exitCode=0 Oct 30 13:25:47.121234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.120428 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" event={"ID":"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5","Type":"ContainerDied","Data":"016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9"} Oct 30 13:25:47.121234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.120473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" event={"ID":"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5","Type":"ContainerDied","Data":"07aae48b7bcca8cdf6cb9d614751be73cba6227b6ecc308ece043b2c154483e5"} Oct 30 13:25:47.121234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.120506 5693 scope.go:122] "RemoveContainer" containerID="016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9" Oct 30 13:25:47.121234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.120671 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz" Oct 30 13:25:47.124095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.124068 5693 generic.go:356] "Generic (PLEG): container finished" podID="1285ceff-bd1a-4d05-9528-63b3328b6002" containerID="d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870" exitCode=0 Oct 30 13:25:47.124285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.124266 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" event={"ID":"1285ceff-bd1a-4d05-9528-63b3328b6002","Type":"ContainerDied","Data":"d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870"} Oct 30 13:25:47.124423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.124407 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" event={"ID":"1285ceff-bd1a-4d05-9528-63b3328b6002","Type":"ContainerDied","Data":"f3e9c76ff16b1ced76d9dd905ccc6fbfa2373411bb845128975aa34e7088f05f"} Oct 30 13:25:47.124590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.124575 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2" Oct 30 13:25:47.166233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.166211 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:47.170555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.170534 5693 scope.go:122] "RemoveContainer" containerID="016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9" Oct 30 13:25:47.170998 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:47.170948 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9\": container with ID starting with 016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9 not found: ID does not exist" containerID="016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9" Oct 30 13:25:47.170998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.170972 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9"} err="failed to get container status \"016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9\": rpc error: code = NotFound desc = could not find container \"016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9\": container with ID starting with 016993ad229e4f8105d39912a7ed8843a59b923d2e946d467b13e95dd46775f9 not found: ID does not exist" Oct 30 13:25:47.170998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.170989 5693 scope.go:122] "RemoveContainer" containerID="d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870" Oct 30 13:25:47.182687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.182659 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume\") pod \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " Oct 30 13:25:47.182806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.182726 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888\" (UniqueName: \"kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888\") pod \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\" (UID: \"cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5\") " Oct 30 13:25:47.183288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.183244 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume" pod "cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" (UID: "cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:47.196357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.196332 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888" pod "cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" (UID: "cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5"). InnerVolumeSpecName "kube-api-access-fr888". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:47.196753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.196734 5693 scope.go:122] "RemoveContainer" containerID="d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870" Oct 30 13:25:47.197126 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:47.197104 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870\": container with ID starting with d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870 not found: ID does not exist" containerID="d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870" Oct 30 13:25:47.197198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.197137 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870"} err="failed to get container status \"d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870\": rpc error: code = NotFound desc = could not find container \"d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870\": container with ID starting with d46317c7bc66ac89abd3c4df4ac9b2a3946c86589b70890fd83e27569f406870 not found: ID does not exist" Oct 30 13:25:47.219853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.219817 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2"] Oct 30 13:25:47.224585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.224557 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ac28e-dd85c49dc-pl4q2"] Oct 30 13:25:47.272794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.272772 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67"] Oct 30 13:25:47.275145 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:25:47.275118 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ff25d3_ae74_4dc3_aee8_9f5ab7def0b0.slice/crio-d01489479e59081a8fa48de2f277777325047c5a1132a09847c0f53586790e7e WatchSource:0}: Error finding container d01489479e59081a8fa48de2f277777325047c5a1132a09847c0f53586790e7e: Status 404 returned error can't find the container with id d01489479e59081a8fa48de2f277777325047c5a1132a09847c0f53586790e7e Oct 30 13:25:47.284851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.284828 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:47.284963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.284852 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr888\" (UniqueName: \"kubernetes.io/projected/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5-kube-api-access-fr888\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:25:47.332759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.332729 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:47.489611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.489560 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:47.491755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.491726 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:25:47.491893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.491773 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:25:47.521915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.521881 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz"] Oct 30 13:25:47.524973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.524946 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8ac28e-7d894f8499-28vhz"] Oct 30 13:25:47.701886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.701789 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:47.853649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:47.853615 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:48.133487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.133448 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" event={"ID":"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0","Type":"ContainerStarted","Data":"4132a037b648ccf393f6beb880e82d2b27e360cc13d8ddd71b1ca252f535f4e4"} Oct 30 13:25:48.133709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.133528 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" event={"ID":"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0","Type":"ContainerStarted","Data":"d01489479e59081a8fa48de2f277777325047c5a1132a09847c0f53586790e7e"} Oct 30 13:25:48.133883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.133844 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:48.135420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.135393 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" event={"ID":"cc06a9a5-c433-4023-95e7-861839c2266f","Type":"ContainerStarted","Data":"8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e"} Oct 30 13:25:48.135545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.135445 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:48.173985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.173927 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" podStartSLOduration=2.173911114 podStartE2EDuration="2.173911114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:48.172024212 +0000 UTC m=+1324.038642386" watchObservedRunningTime="2026-10-30 13:25:48.173911114 +0000 UTC m=+1324.040529291" Oct 30 13:25:48.210843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.210786 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" podStartSLOduration=2.21077329 podStartE2EDuration="2.21077329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:48.210736126 +0000 UTC m=+1324.077354301" watchObservedRunningTime="2026-10-30 13:25:48.21077329 +0000 UTC m=+1324.077391459" Oct 30 13:25:48.255725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.255685 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1285ceff-bd1a-4d05-9528-63b3328b6002" path="/var/lib/kubelet/pods/1285ceff-bd1a-4d05-9528-63b3328b6002/volumes" Oct 30 13:25:48.256227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:48.256160 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" path="/var/lib/kubelet/pods/cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5/volumes" Oct 30 13:25:48.735640 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:48.735613 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccafbd8_9c8c_4493_8954_d5a5813391e0.slice/crio-70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:25:48.735786 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:48.735679 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccafbd8_9c8c_4493_8954_d5a5813391e0.slice/crio-conmon-70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:25:49.145985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.145952 5693 generic.go:356] "Generic (PLEG): container finished" podID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerID="70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7" exitCode=0 Oct 30 13:25:49.146174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.146025 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" event={"ID":"accafbd8-9c8c-4493-8954-d5a5813391e0","Type":"ContainerDied","Data":"70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7"} Oct 30 13:25:49.146722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.146706 5693 scope.go:122] "RemoveContainer" containerID="70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7" Oct 30 13:25:49.148015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.147987 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:25:49.148080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.148053 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:25:49.148116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.148084 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:25:49.221325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:49.221287 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:50.159971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:50.159942 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" event={"ID":"accafbd8-9c8c-4493-8954-d5a5813391e0","Type":"ContainerStarted","Data":"a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb"} Oct 30 13:25:50.665220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:50.665185 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:50.665404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:50.665366 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 13:25:50.665450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:50.665410 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 13:25:50.824300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:50.824267 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:51.665389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:51.665351 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 13:25:51.665827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:51.665404 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 13:25:52.054292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.054255 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:52.204768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.204731 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:52.449481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.449397 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:52.650542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.650506 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:52.677622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.666386 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 13:25:52.677622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.666458 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 13:25:52.796665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.796621 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5"] Oct 30 13:25:52.888860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:52.888814 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:53.084618 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:25:53.084514 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:25:53.508417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.508380 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh"] Oct 30 13:25:53.509980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.509945 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1285ceff-bd1a-4d05-9528-63b3328b6002" containerName="maas-api" Oct 30 13:25:53.509980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.509973 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1285ceff-bd1a-4d05-9528-63b3328b6002" containerName="maas-api" Oct 30 13:25:53.510180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.509994 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" containerName="payload-pre-processing" Oct 30 13:25:53.510180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.510013 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" containerName="payload-pre-processing" Oct 30 13:25:53.512395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.512361 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1285ceff-bd1a-4d05-9528-63b3328b6002" containerName="maas-api" Oct 30 13:25:53.512584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.512568 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb17be3b-dd0d-4bd5-8c60-daae72a3ebb5" containerName="payload-pre-processing" Oct 30 13:25:53.562872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.562839 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.563142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.562994 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh"] Oct 30 13:25:53.576385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.576355 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:53.579570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.579547 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-5cee02\"/\"openshift-service-ca.crt\"" Oct 30 13:25:53.579698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.579678 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-5cee02\"/\"kube-root-ca.crt\"" Oct 30 13:25:53.591784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.591762 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-5cee02\"/\"default-dockercfg-zsjz6\"" Oct 30 13:25:53.591899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.591883 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-5cee02\"/\"praxis-route-5cee02-kserve-self-signed-certs\"" Oct 30 13:25:53.664913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.664879 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 13:25:53.665091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.664959 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 13:25:53.667708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.667667 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.667844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.667742 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.667844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.667802 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst2x\" (UniqueName: \"kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.667931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.667879 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.667985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.667973 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.668023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.667993 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.736623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.730067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.771806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.771890 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.771970 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.772061 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.772088 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst2x\" (UniqueName: \"kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.772203 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.773293 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.773868 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.777668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.775924 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.801640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.783315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.801640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.783949 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.824628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.817790 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst2x\" (UniqueName: \"kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x\") pod \"praxis-route-5cee02-kserve-5fc5dfcf57-92wsh\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:53.886908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:53.886861 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:54.220044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:54.220018 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh"] Oct 30 13:25:54.222009 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:25:54.221978 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd14428e4_2205_4a73_92f7_a2afbf0d9140.slice/crio-ced4fe7b4d6c27c927c28d96a7c09a8728cf9fe000f7cef829a4427334089d3d WatchSource:0}: Error finding container ced4fe7b4d6c27c927c28d96a7c09a8728cf9fe000f7cef829a4427334089d3d: Status 404 returned error can't find the container with id ced4fe7b4d6c27c927c28d96a7c09a8728cf9fe000f7cef829a4427334089d3d Oct 30 13:25:54.504941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:54.504902 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:54.664608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:54.664557 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 13:25:54.664771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:54.664632 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 13:25:55.197569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.197534 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" event={"ID":"d14428e4-2205-4a73-92f7-a2afbf0d9140","Type":"ContainerStarted","Data":"922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372"} Oct 30 13:25:55.198046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.197579 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" event={"ID":"d14428e4-2205-4a73-92f7-a2afbf0d9140","Type":"ContainerStarted","Data":"ced4fe7b4d6c27c927c28d96a7c09a8728cf9fe000f7cef829a4427334089d3d"} Oct 30 13:25:55.669292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.669261 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:55.669652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.669626 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:55.670350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.670331 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:25:55.730619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.710931 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podStartSLOduration=2.710915344 podStartE2EDuration="2.710915344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:55.248298851 +0000 UTC m=+1331.114917039" watchObservedRunningTime="2026-10-30 13:25:55.710915344 +0000 UTC m=+1331.577533518" Oct 30 13:25:55.887268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:55.887235 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:25:56.221234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:56.221200 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:57.933525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:57.933491 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:59.149052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:59.149021 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:25:59.149431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:59.149094 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:25:59.644588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:25:59.644548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:01.815098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.815060 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5"] Oct 30 13:26:01.820581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.820557 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:01.849644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.848685 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d797fc-serving-cert\"" Oct 30 13:26:01.852917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.852884 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5"] Oct 30 13:26:01.933875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.933835 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:01.934045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.933891 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:01.934130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.934109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:01.934168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.934158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vbq5\" (UniqueName: \"kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:01.949476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:01.949432 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.036221 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.036291 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vbq5\" (UniqueName: \"kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.036482 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.036526 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:26:02.037579 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-serving-cert: secret "maas-api-e2e-praxis-a-d797fc-serving-cert" not found Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:26:02.037664 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls podName:1ea6e696-7f58-4427-b35b-b59014a4bd9a nodeName:}" failed. No retries permitted until 2026-10-30 13:26:02.537644562 +0000 UTC m=+1338.404262730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls") pod "maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" (UID: "1ea6e696-7f58-4427-b35b-b59014a4bd9a") : secret "maas-api-e2e-praxis-a-d797fc-serving-cert" not found Oct 30 13:26:02.038282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.037976 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.046577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.046504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.086013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.085933 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vbq5\" (UniqueName: \"kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.100199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.100161 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.235077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.235042 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.543914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.543873 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.546364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.546333 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:02.739967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:02.739931 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:03.121975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:03.121900 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5"] Oct 30 13:26:03.124841 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:26:03.124805 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea6e696_7f58_4427_b35b_b59014a4bd9a.slice/crio-a1163647796e6d4730307646a22669823e0c22b635d1df932c18ba088208dfe2 WatchSource:0}: Error finding container a1163647796e6d4730307646a22669823e0c22b635d1df932c18ba088208dfe2: Status 404 returned error can't find the container with id a1163647796e6d4730307646a22669823e0c22b635d1df932c18ba088208dfe2 Oct 30 13:26:03.262245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:03.262206 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" event={"ID":"1ea6e696-7f58-4427-b35b-b59014a4bd9a","Type":"ContainerStarted","Data":"a1163647796e6d4730307646a22669823e0c22b635d1df932c18ba088208dfe2"} Oct 30 13:26:03.746085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:03.746045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:03.888147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:03.888115 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:26:03.901310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:03.901284 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:26:04.269929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:04.269895 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" event={"ID":"1ea6e696-7f58-4427-b35b-b59014a4bd9a","Type":"ContainerStarted","Data":"f98c287b4ca54380ba8ff871532d333b03217ecfbb6ebf2e4029023aeef1deeb"} Oct 30 13:26:04.270375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:04.269970 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:04.282249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:04.282226 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:26:04.309372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:04.309328 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" podStartSLOduration=3.022603749 podStartE2EDuration="3.309313257s" totalImagesPullingTime="286.709508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:04.306854195 +0000 UTC m=+1340.173472369" watchObservedRunningTime="2026-10-30 13:26:04.309313257 +0000 UTC m=+1340.175931430" Oct 30 13:26:05.535622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:05.522654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:07.095810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:07.095778 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:07.892028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:07.891988 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" podUID="d643a9af-c04e-40b4-b515-843e51324084" containerName="main" containerID="cri-o://239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e" gracePeriod=45 Oct 30 13:26:07.896650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:07.896612 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" podUID="d643a9af-c04e-40b4-b515-843e51324084" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": dial tcp 10.134.0.127:8000: connect: connection refused" Oct 30 13:26:08.171172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.171145 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:26:08.257618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.257579 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs\" (UniqueName: \"kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs\") pod \"d643a9af-c04e-40b4-b515-843e51324084\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " Oct 30 13:26:08.257798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.257687 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home\") pod \"d643a9af-c04e-40b4-b515-843e51324084\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " Oct 30 13:26:08.257798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.257734 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq\" (UniqueName: \"kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq\") pod \"d643a9af-c04e-40b4-b515-843e51324084\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " Oct 30 13:26:08.257798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.257790 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache\") pod \"d643a9af-c04e-40b4-b515-843e51324084\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " Oct 30 13:26:08.257988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.257841 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir\") pod \"d643a9af-c04e-40b4-b515-843e51324084\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " Oct 30 13:26:08.258052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258025 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm\") pod \"d643a9af-c04e-40b4-b515-843e51324084\" (UID: \"d643a9af-c04e-40b4-b515-843e51324084\") " Oct 30 13:26:08.258199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258121 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache" pod "d643a9af-c04e-40b4-b515-843e51324084" (UID: "d643a9af-c04e-40b4-b515-843e51324084"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:08.258331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258240 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir" pod "d643a9af-c04e-40b4-b515-843e51324084" (UID: "d643a9af-c04e-40b4-b515-843e51324084"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:08.258331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258297 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home" pod "d643a9af-c04e-40b4-b515-843e51324084" (UID: "d643a9af-c04e-40b4-b515-843e51324084"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:08.258634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258581 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.258787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258771 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.258890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.258878 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.260714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.260674 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq" pod "d643a9af-c04e-40b4-b515-843e51324084" (UID: "d643a9af-c04e-40b4-b515-843e51324084"). InnerVolumeSpecName "kube-api-access-vxnnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:08.260818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.260771 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs" pod "d643a9af-c04e-40b4-b515-843e51324084" (UID: "d643a9af-c04e-40b4-b515-843e51324084"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:08.260999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.260979 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm" pod "d643a9af-c04e-40b4-b515-843e51324084" (UID: "d643a9af-c04e-40b4-b515-843e51324084"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:08.299823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.299790 5693 generic.go:356] "Generic (PLEG): container finished" podID="d643a9af-c04e-40b4-b515-843e51324084" containerID="239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e" exitCode=0 Oct 30 13:26:08.299823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.299827 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" Oct 30 13:26:08.300058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.299866 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" event={"ID":"d643a9af-c04e-40b4-b515-843e51324084","Type":"ContainerDied","Data":"239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e"} Oct 30 13:26:08.300058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.299919 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5" event={"ID":"d643a9af-c04e-40b4-b515-843e51324084","Type":"ContainerDied","Data":"ccf32028b078c82b6c5af6617c9283dea52c2c0167c00a8b53641c26b76777b7"} Oct 30 13:26:08.300058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.299944 5693 scope.go:122] "RemoveContainer" containerID="239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e" Oct 30 13:26:08.310258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.310235 5693 scope.go:122] "RemoveContainer" containerID="239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e" Oct 30 13:26:08.310536 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:26:08.310516 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e\": container with ID starting with 239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e not found: ID does not exist" containerID="239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e" Oct 30 13:26:08.310650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.310542 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e"} err="failed to get container status \"239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e\": rpc error: code = NotFound desc = could not find container \"239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e\": container with ID starting with 239eaaf3ba4d6ee153c635113c445176d542f63d04e41899e97db936d271416e not found: ID does not exist" Oct 30 13:26:08.355310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.355276 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5"] Oct 30 13:26:08.358612 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.358575 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8ac28e/ipp-route-8ac28e-kserve-7c5c974c49-2njc5"] Oct 30 13:26:08.360515 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.360426 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxnnq\" (UniqueName: \"kubernetes.io/projected/d643a9af-c04e-40b4-b515-843e51324084-kube-api-access-vxnnq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.360515 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.360453 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d643a9af-c04e-40b4-b515-843e51324084-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.360515 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.360471 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d643a9af-c04e-40b4-b515-843e51324084-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.693244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:08.693007 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:10.255436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:10.255401 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d643a9af-c04e-40b4-b515-843e51324084" path="/var/lib/kubelet/pods/d643a9af-c04e-40b4-b515-843e51324084/volumes" Oct 30 13:26:10.284739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:10.284709 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:26:10.298498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:10.298465 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:11.864135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:11.864103 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:13.314498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:13.314467 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:13.847738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:13.847706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:14.003694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:14.003660 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:14.160680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:14.160562 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:14.535539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:14.535507 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:15.905166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:15.905129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:17.540808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:17.540776 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:19.730307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:19.730252 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:20.899950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:20.899899 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:21.067873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:21.067813 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:21.289365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:21.289329 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:21.315105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:21.308671 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb"] Oct 30 13:26:21.315671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:21.315632 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" containerID="cri-o://38043d289ecfab34058b20967cb67e150501fe6fe1162a13c3ff8ded1a70d871" gracePeriod=30 Oct 30 13:26:21.565847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:21.565175 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.240419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.240373 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.750918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.750887 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp"] Oct 30 13:26:22.751930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.751912 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d643a9af-c04e-40b4-b515-843e51324084" containerName="main" Oct 30 13:26:22.752007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.751933 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d643a9af-c04e-40b4-b515-843e51324084" containerName="main" Oct 30 13:26:22.752078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.752067 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d643a9af-c04e-40b4-b515-843e51324084" containerName="main" Oct 30 13:26:22.757440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.757417 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.778221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.778200 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-65e250-openshift-default-dockercfg-4v977\"" Oct 30 13:26:22.798722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.798680 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp"] Oct 30 13:26:22.834831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.834789 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.834949 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835007 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835076 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835122 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835141 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835208 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btcn\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835249 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.835514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.835404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.839301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.839279 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.943628 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btcn\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.943689 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.943824 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.943853 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.944070 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.944136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.944165 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.944230 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.944260 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.944751 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.953619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.948925 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.960621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.959433 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.960621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.959728 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.960621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.959939 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.983623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.981176 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:22.993562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.993523 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.995150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.994959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btcn\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:23.002663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:22.997873 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:23.021648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.006752 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token\") pod \"e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:23.074093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.074053 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:23.087523 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:26:23.086854 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:26:23.459430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.440877 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp"] Oct 30 13:26:23.468732 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:26:23.467771 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b065a33_cc37_42bd_b49b_cb1bff218923.slice/crio-4c2a63a06f22f045098eefc3a86b7fe7e93f0dfedda2823bf7c30dc967c824b5 WatchSource:0}: Error finding container 4c2a63a06f22f045098eefc3a86b7fe7e93f0dfedda2823bf7c30dc967c824b5: Status 404 returned error can't find the container with id 4c2a63a06f22f045098eefc3a86b7fe7e93f0dfedda2823bf7c30dc967c824b5 Oct 30 13:26:23.478425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.478379 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:23.478573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.478476 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:23.478573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.478523 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:23.552649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.552564 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" event={"ID":"7b065a33-cc37-42bd-b49b-cb1bff218923","Type":"ContainerStarted","Data":"4c2a63a06f22f045098eefc3a86b7fe7e93f0dfedda2823bf7c30dc967c824b5"} Oct 30 13:26:23.686621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:23.679225 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:24.065696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:24.065656 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:24.570156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:24.570107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" event={"ID":"7b065a33-cc37-42bd-b49b-cb1bff218923","Type":"ContainerStarted","Data":"e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6"} Oct 30 13:26:24.642452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:24.642368 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podStartSLOduration=2.642345375 podStartE2EDuration="2.642345375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:24.62616353 +0000 UTC m=+1360.492781705" watchObservedRunningTime="2026-10-30 13:26:24.642345375 +0000 UTC m=+1360.508963549" Oct 30 13:26:25.083661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.079977 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:25.088949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.088915 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:25.293624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.289141 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:25.608659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.601369 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:25.645727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.643099 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.807986 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t"] Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.820354 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.822559 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t"] Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.822692 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873128 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873292 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873363 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873400 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873468 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkph\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873643 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.883671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.873675 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:25.985076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:25.930204 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b35c0f-openshift-default-dockercfg-47v4x\"" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.016893 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.016949 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.016993 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.017034 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.017141 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.017213 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.017252 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.017329 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkph\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.017391 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.018582 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.019521 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.032160 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.032689 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.032983 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.108703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.101932 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.256674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.235206 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:26.391428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.334977 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.435622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.428766 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.458415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.458346 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkph\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.505637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.489129 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token\") pod \"e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:26.690623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.671860 5693 generic.go:356] "Generic (PLEG): container finished" podID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerID="38043d289ecfab34058b20967cb67e150501fe6fe1162a13c3ff8ded1a70d871" exitCode=0 Oct 30 13:26:26.690623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.677551 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" event={"ID":"c466e2e1-ee81-4508-ae95-1c0e56fdef9a","Type":"ContainerDied","Data":"38043d289ecfab34058b20967cb67e150501fe6fe1162a13c3ff8ded1a70d871"} Oct 30 13:26:26.690623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.677686 5693 scope.go:122] "RemoveContainer" containerID="884e6297f64f7ccd8626b47b74b313623747fbd44341d64ecf1b45f3b043bbb3" Oct 30 13:26:26.725415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:26.721695 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:27.107470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.107437 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:26:27.109529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.109494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:27.200527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200496 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.200743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200695 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.200820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200744 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.200820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200799 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.200929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200840 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.200929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200900 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.201031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.200959 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.201085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.201065 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.201172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.201149 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo\") pod \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\" (UID: \"c466e2e1-ee81-4508-ae95-1c0e56fdef9a\") " Oct 30 13:26:27.202196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.201963 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.202570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.202380 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.203223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.203016 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.203223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.203184 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:27.203626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.203585 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.209347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.209182 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:27.209347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.209320 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "kube-api-access-54ftw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:27.210868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.210811 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.211236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.211209 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo" pod "c466e2e1-ee81-4508-ae95-1c0e56fdef9a" (UID: "c466e2e1-ee81-4508-ae95-1c0e56fdef9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:27.303051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303017 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54ftw\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-kube-api-access-54ftw\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303044 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303055 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303063 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303071 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303079 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303088 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303095 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.303327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.303102 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c466e2e1-ee81-4508-ae95-1c0e56fdef9a-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.323818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.323784 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:27.470645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.470620 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t"] Oct 30 13:26:27.473521 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:26:27.473488 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e947291_0a68_48df_b65a_d3eeda038447.slice/crio-eca115ab4e39f779b9b8c94025df12d189e754eff0cfebb2a9d48fec818792df WatchSource:0}: Error finding container eca115ab4e39f779b9b8c94025df12d189e754eff0cfebb2a9d48fec818792df: Status 404 returned error can't find the container with id eca115ab4e39f779b9b8c94025df12d189e754eff0cfebb2a9d48fec818792df Oct 30 13:26:27.476535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.476496 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:27.476635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.476567 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:27.476635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.476628 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:27.682715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.682677 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" event={"ID":"4e947291-0a68-48df-b65a-d3eeda038447","Type":"ContainerStarted","Data":"ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9"} Oct 30 13:26:27.682910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.682739 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" event={"ID":"4e947291-0a68-48df-b65a-d3eeda038447","Type":"ContainerStarted","Data":"eca115ab4e39f779b9b8c94025df12d189e754eff0cfebb2a9d48fec818792df"} Oct 30 13:26:27.684573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.684549 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" Oct 30 13:26:27.684729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.684589 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb" event={"ID":"c466e2e1-ee81-4508-ae95-1c0e56fdef9a","Type":"ContainerDied","Data":"3b1ea83d0fc0e91c513726cf9f5e6480d22d2aa777addc8cd35872e9aaf9da7a"} Oct 30 13:26:27.684729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.684654 5693 scope.go:122] "RemoveContainer" containerID="38043d289ecfab34058b20967cb67e150501fe6fe1162a13c3ff8ded1a70d871" Oct 30 13:26:27.723517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.723454 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:27.750621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.744873 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podStartSLOduration=2.744854558 podStartE2EDuration="2.744854558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:27.732040485 +0000 UTC m=+1363.598658670" watchObservedRunningTime="2026-10-30 13:26:27.744854558 +0000 UTC m=+1363.611472734" Oct 30 13:26:27.780054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.780020 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb"] Oct 30 13:26:27.785464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:27.785436 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e3e617-openshift-default-5d5fdbc55c-v29wb"] Oct 30 13:26:28.263045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:28.263009 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" path="/var/lib/kubelet/pods/c466e2e1-ee81-4508-ae95-1c0e56fdef9a/volumes" Oct 30 13:26:28.746202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:28.746170 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:29.157680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.157261 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk"] Oct 30 13:26:29.158328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.158297 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" Oct 30 13:26:29.158328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.158326 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" Oct 30 13:26:29.158549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.158370 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" Oct 30 13:26:29.158549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.158379 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" Oct 30 13:26:29.158549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.158518 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" Oct 30 13:26:29.158549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.158532 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c466e2e1-ee81-4508-ae95-1c0e56fdef9a" containerName="istio-proxy" Oct 30 13:26:29.164086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.164060 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.167434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.167404 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk"] Oct 30 13:26:29.178793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.178758 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-65e250\"" Oct 30 13:26:29.315685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.313786 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-65e250-dockercfg-fkfjk\"" Oct 30 13:26:29.428348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.423580 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.428348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.423939 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg\") pod \"payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.525924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.525886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.526091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.525980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg\") pod \"payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.528645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.526846 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.570872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.570835 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg\") pod \"payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:29.714443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.712804 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:29.716108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.716085 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:26:29.781753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:29.781708 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:30.117082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:30.117050 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk"] Oct 30 13:26:30.121016 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:26:30.120985 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dfc294_6cac_4419_82b3_93a5b2f11a19.slice/crio-4d0070d695155cd6be6666b23192b7c00d173fbbb6fab9ffba074368ca7b7182 WatchSource:0}: Error finding container 4d0070d695155cd6be6666b23192b7c00d173fbbb6fab9ffba074368ca7b7182: Status 404 returned error can't find the container with id 4d0070d695155cd6be6666b23192b7c00d173fbbb6fab9ffba074368ca7b7182 Oct 30 13:26:30.720006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:30.719969 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" event={"ID":"c5dfc294-6cac-4419-82b3-93a5b2f11a19","Type":"ContainerStarted","Data":"36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264"} Oct 30 13:26:30.720232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:30.720026 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" event={"ID":"c5dfc294-6cac-4419-82b3-93a5b2f11a19","Type":"ContainerStarted","Data":"4d0070d695155cd6be6666b23192b7c00d173fbbb6fab9ffba074368ca7b7182"} Oct 30 13:26:30.720232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:30.720200 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:30.759285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:30.759213 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" podStartSLOduration=1.759188875 podStartE2EDuration="1.759188875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:30.758032337 +0000 UTC m=+1366.624650510" watchObservedRunningTime="2026-10-30 13:26:30.759188875 +0000 UTC m=+1366.625807106" Oct 30 13:26:31.253648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:31.253589 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:31.406289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:31.406253 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:31.922023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:31.921948 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:31.991996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:31.991963 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7"] Oct 30 13:26:31.996812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:31.996784 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.024905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.024867 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7"] Oct 30 13:26:32.025379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.025356 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b35c0f\"" Oct 30 13:26:32.025837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.025818 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b35c0f-dockercfg-tvlx5\"" Oct 30 13:26:32.060615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.057643 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn"] Oct 30 13:26:32.068443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.068418 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn"] Oct 30 13:26:32.068823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.068787 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.071116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.071089 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:32.189673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.188748 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.189673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.188816 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcssc\" (UniqueName: \"kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc\") pod \"payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.189673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.188865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zt9\" (UniqueName: \"kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9\") pod \"payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.189673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.188903 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.233932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.230737 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:32.328283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.316135 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.328283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.316202 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcssc\" (UniqueName: \"kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc\") pod \"payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.328283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.316243 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zt9\" (UniqueName: \"kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9\") pod \"payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.328283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.316275 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.328283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.317319 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.328283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.318389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.344882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.343138 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcssc\" (UniqueName: \"kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc\") pod \"payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.345941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.345916 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zt9\" (UniqueName: \"kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9\") pod \"payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.422873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.422835 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:32.614147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.614111 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:32.747025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.746997 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn"] Oct 30 13:26:32.879855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.879804 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:32.953616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:32.953514 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7"] Oct 30 13:26:32.956862 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:26:32.956827 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafecc174_c237_4352_a666_a7b3b56b5428.slice/crio-ba73c1e46a095fe6c09d224b3353722d46605a312266ba399b3bd01c2788ffda WatchSource:0}: Error finding container ba73c1e46a095fe6c09d224b3353722d46605a312266ba399b3bd01c2788ffda: Status 404 returned error can't find the container with id ba73c1e46a095fe6c09d224b3353722d46605a312266ba399b3bd01c2788ffda Oct 30 13:26:33.749947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.749907 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" event={"ID":"afecc174-c237-4352-a666-a7b3b56b5428","Type":"ContainerStarted","Data":"241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c"} Oct 30 13:26:33.750587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.749976 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" event={"ID":"afecc174-c237-4352-a666-a7b3b56b5428","Type":"ContainerStarted","Data":"ba73c1e46a095fe6c09d224b3353722d46605a312266ba399b3bd01c2788ffda"} Oct 30 13:26:33.750949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.750925 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:33.753501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.753476 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" event={"ID":"90c5979c-50ca-4258-bf72-01336e03314f","Type":"ContainerStarted","Data":"050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e"} Oct 30 13:26:33.753717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.753694 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" event={"ID":"90c5979c-50ca-4258-bf72-01336e03314f","Type":"ContainerStarted","Data":"6e2c635a9f9c0b9e928288de5a64b2f64a41254b086bc371137e4f903adf5785"} Oct 30 13:26:33.753801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.753737 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:33.794147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.794097 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" podStartSLOduration=2.794085049 podStartE2EDuration="2.794085049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:33.790788562 +0000 UTC m=+1369.657406729" watchObservedRunningTime="2026-10-30 13:26:33.794085049 +0000 UTC m=+1369.660703221" Oct 30 13:26:33.828368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:33.828326 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" podStartSLOduration=1.828313968 podStartE2EDuration="1.828313968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:33.827415979 +0000 UTC m=+1369.694034152" watchObservedRunningTime="2026-10-30 13:26:33.828313968 +0000 UTC m=+1369.694932141" Oct 30 13:26:34.465313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:34.465235 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:35.773152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:35.773114 5693 generic.go:356] "Generic (PLEG): container finished" podID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerID="ea60ef3bb25e9934a7526f1bec552506158275661eb2419b9d7a5f5c3b0698d5" exitCode=0 Oct 30 13:26:35.773750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:35.773187 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" event={"ID":"cc479868-520a-45f0-bc7c-ba06a8f95830","Type":"ContainerDied","Data":"ea60ef3bb25e9934a7526f1bec552506158275661eb2419b9d7a5f5c3b0698d5"} Oct 30 13:26:35.773921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:35.773899 5693 scope.go:122] "RemoveContainer" containerID="ea60ef3bb25e9934a7526f1bec552506158275661eb2419b9d7a5f5c3b0698d5" Oct 30 13:26:35.776220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:35.776187 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:35.776361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:35.776277 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:35.776361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:35.776323 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:26:36.062679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:36.059247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:36.781736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:36.781698 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" event={"ID":"cc479868-520a-45f0-bc7c-ba06a8f95830","Type":"ContainerStarted","Data":"2e84c545b2550cebba9716a6c9f8217be03ba36bfeb9f8226e5e3cb9a4916233"} Oct 30 13:26:37.696077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:37.696039 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:26:37.697550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:37.697522 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 13:26:37.697697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:37.697584 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 13:26:37.804490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:37.804456 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.696318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:38.696284 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 13:26:38.696493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:38.696337 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 13:26:38.947577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:38.947453 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.095303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:39.095267 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.517870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:39.517814 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.696235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:39.696192 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 13:26:39.696405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:39.696263 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 13:26:40.208629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:40.208582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:40.696181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:40.696145 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 13:26:40.696367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:40.696214 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 13:26:41.695494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:41.695456 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 13:26:41.695916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:41.695520 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 13:26:41.748998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:41.748964 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:26:42.008637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:41.999098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:42.026398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:42.026362 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:26:42.700213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:42.700179 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:26:42.830031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:42.829995 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:26:43.701882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:43.701848 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:44.763112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:44.763081 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:26:45.396679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:45.396643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:45.545119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:45.545080 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:45.682521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:45.682424 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:45.776286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:45.776260 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:26:47.627977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:47.627839 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:49.602708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:49.602666 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:51.064362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.064298 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg"] Oct 30 13:26:51.070942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.070919 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.089690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.089655 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg"] Oct 30 13:26:51.098198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.098141 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b35c0f-serving-cert\"" Oct 30 13:26:51.156434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.156399 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.156606 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.156508 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfh7\" (UniqueName: \"kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.156606 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.156551 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.156704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.156620 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.209650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.209588 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:51.258007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.257974 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.258199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.258062 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfh7\" (UniqueName: \"kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.258199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.258111 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.258199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.258185 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.258817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.258792 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.260504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.260476 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.260672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.260647 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.276762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.276742 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfh7\" (UniqueName: \"kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7\") pod \"maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.357683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.356328 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:51.393628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.391901 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:51.723147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.723123 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg"] Oct 30 13:26:51.725432 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:26:51.725401 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3e6d59_ddc6_4b7a_8df4_ec475f8dd796.slice/crio-c0af160cfa6aa65042706fd5b021fb0ccfbc2a0c0cc15a206fefee41a452d8c1 WatchSource:0}: Error finding container c0af160cfa6aa65042706fd5b021fb0ccfbc2a0c0cc15a206fefee41a452d8c1: Status 404 returned error can't find the container with id c0af160cfa6aa65042706fd5b021fb0ccfbc2a0c0cc15a206fefee41a452d8c1 Oct 30 13:26:51.934405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:51.934368 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" event={"ID":"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796","Type":"ContainerStarted","Data":"c0af160cfa6aa65042706fd5b021fb0ccfbc2a0c0cc15a206fefee41a452d8c1"} Oct 30 13:26:53.014987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:53.004197 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:53.032642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:53.030052 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" event={"ID":"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796","Type":"ContainerStarted","Data":"9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07"} Oct 30 13:26:53.032642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:53.030263 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:53.064011 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:26:53.063971 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:26:53.072656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:53.072593 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" podStartSLOduration=1.607455051 podStartE2EDuration="2.072579491s" totalImagesPullingTime="465.12444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:53.070147599 +0000 UTC m=+1388.936765792" watchObservedRunningTime="2026-10-30 13:26:53.072579491 +0000 UTC m=+1388.939197664" Oct 30 13:26:53.704239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:53.704205 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:54.768620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:54.762384 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:56.588625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:56.585123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:58.345626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:58.342448 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:59.042271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.042244 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:26:59.762800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.762761 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm"] Oct 30 13:26:59.763305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.763058 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" podUID="96da0066-a10e-4b12-9284-8635da91d592" containerName="maas-api" containerID="cri-o://c6eaa386651749a18440b4dd6a52eeae42b120998c002399f160442ace67e7ad" gracePeriod=30 Oct 30 13:26:59.890690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.890653 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:59.955046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.955007 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk"] Oct 30 13:26:59.955288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.955259 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" podUID="439e921f-9e85-4bcd-a458-a7de27e0ca70" containerName="payload-processing" containerID="cri-o://fb5e00cfbc9828b2d137ba8906106bb9e907a860992c60f02ddafbc9738bf719" gracePeriod=30 Oct 30 13:26:59.978697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:26:59.978650 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp"] Oct 30 13:27:00.081012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.080983 5693 generic.go:356] "Generic (PLEG): container finished" podID="96da0066-a10e-4b12-9284-8635da91d592" containerID="c6eaa386651749a18440b4dd6a52eeae42b120998c002399f160442ace67e7ad" exitCode=0 Oct 30 13:27:00.081118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.081055 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" event={"ID":"96da0066-a10e-4b12-9284-8635da91d592","Type":"ContainerDied","Data":"c6eaa386651749a18440b4dd6a52eeae42b120998c002399f160442ace67e7ad"} Oct 30 13:27:00.081170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.081115 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" event={"ID":"96da0066-a10e-4b12-9284-8635da91d592","Type":"ContainerDied","Data":"b4f6d0a8b25fede2aacc6484c3726bbca1a478e761801d167a61886b0173a78f"} Oct 30 13:27:00.081170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.081138 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f6d0a8b25fede2aacc6484c3726bbca1a478e761801d167a61886b0173a78f" Oct 30 13:27:00.082717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.082692 5693 generic.go:356] "Generic (PLEG): container finished" podID="439e921f-9e85-4bcd-a458-a7de27e0ca70" containerID="fb5e00cfbc9828b2d137ba8906106bb9e907a860992c60f02ddafbc9738bf719" exitCode=0 Oct 30 13:27:00.082825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.082759 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" event={"ID":"439e921f-9e85-4bcd-a458-a7de27e0ca70","Type":"ContainerDied","Data":"fb5e00cfbc9828b2d137ba8906106bb9e907a860992c60f02ddafbc9738bf719"} Oct 30 13:27:00.082936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.082920 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" podUID="74bb88af-59ff-426d-b070-c9a4e8944e88" containerName="payload-pre-processing" containerID="cri-o://eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3" gracePeriod=30 Oct 30 13:27:00.119645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.119586 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:00.147212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.147181 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:27:00.217617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.217568 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:27:00.228375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.228343 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j\" (UniqueName: \"kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j\") pod \"96da0066-a10e-4b12-9284-8635da91d592\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " Oct 30 13:27:00.228542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.228378 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs\") pod \"96da0066-a10e-4b12-9284-8635da91d592\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " Oct 30 13:27:00.228542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.228464 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls\") pod \"96da0066-a10e-4b12-9284-8635da91d592\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " Oct 30 13:27:00.228542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.228527 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca\") pod \"96da0066-a10e-4b12-9284-8635da91d592\" (UID: \"96da0066-a10e-4b12-9284-8635da91d592\") " Oct 30 13:27:00.229338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.229313 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca" pod "96da0066-a10e-4b12-9284-8635da91d592" (UID: "96da0066-a10e-4b12-9284-8635da91d592"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:00.231017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.230988 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j" pod "96da0066-a10e-4b12-9284-8635da91d592" (UID: "96da0066-a10e-4b12-9284-8635da91d592"). InnerVolumeSpecName "kube-api-access-6gv5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:00.231309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.231268 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs" pod "96da0066-a10e-4b12-9284-8635da91d592" (UID: "96da0066-a10e-4b12-9284-8635da91d592"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:00.241783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.241752 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls" pod "96da0066-a10e-4b12-9284-8635da91d592" (UID: "96da0066-a10e-4b12-9284-8635da91d592"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:00.338676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.333853 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume\") pod \"439e921f-9e85-4bcd-a458-a7de27e0ca70\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " Oct 30 13:27:00.338676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.333913 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk\" (UniqueName: \"kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk\") pod \"439e921f-9e85-4bcd-a458-a7de27e0ca70\" (UID: \"439e921f-9e85-4bcd-a458-a7de27e0ca70\") " Oct 30 13:27:00.338676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.334504 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gv5j\" (UniqueName: \"kubernetes.io/projected/96da0066-a10e-4b12-9284-8635da91d592-kube-api-access-6gv5j\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.338676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.334529 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.338676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.334545 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96da0066-a10e-4b12-9284-8635da91d592-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.338676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.334559 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96da0066-a10e-4b12-9284-8635da91d592-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.395252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.395196 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume" pod "439e921f-9e85-4bcd-a458-a7de27e0ca70" (UID: "439e921f-9e85-4bcd-a458-a7de27e0ca70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:00.409621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.395808 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:00.434627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.418857 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk" pod "439e921f-9e85-4bcd-a458-a7de27e0ca70" (UID: "439e921f-9e85-4bcd-a458-a7de27e0ca70"). InnerVolumeSpecName "kube-api-access-g6hxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:00.456630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.452529 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e921f-9e85-4bcd-a458-a7de27e0ca70-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.456630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.452579 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hxk\" (UniqueName: \"kubernetes.io/projected/439e921f-9e85-4bcd-a458-a7de27e0ca70-kube-api-access-g6hxk\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.540042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.539652 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:27:00.584274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.584240 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:00.655445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.655354 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89\" (UniqueName: \"kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89\") pod \"74bb88af-59ff-426d-b070-c9a4e8944e88\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " Oct 30 13:27:00.655672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.655517 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume\") pod \"74bb88af-59ff-426d-b070-c9a4e8944e88\" (UID: \"74bb88af-59ff-426d-b070-c9a4e8944e88\") " Oct 30 13:27:00.655941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.655917 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume" pod "74bb88af-59ff-426d-b070-c9a4e8944e88" (UID: "74bb88af-59ff-426d-b070-c9a4e8944e88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:00.656184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.656164 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb88af-59ff-426d-b070-c9a4e8944e88-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.657489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.657472 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89" pod "74bb88af-59ff-426d-b070-c9a4e8944e88" (UID: "74bb88af-59ff-426d-b070-c9a4e8944e88"). InnerVolumeSpecName "kube-api-access-jlw89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:00.757313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.757275 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlw89\" (UniqueName: \"kubernetes.io/projected/74bb88af-59ff-426d-b070-c9a4e8944e88-kube-api-access-jlw89\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.768476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.768448 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:00.963079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:00.962959 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:01.091359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.091332 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" Oct 30 13:27:01.091541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.091366 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk" event={"ID":"439e921f-9e85-4bcd-a458-a7de27e0ca70","Type":"ContainerDied","Data":"118ac00c5ed22227d837819b8374b6c49079397d7b30e17a377f01ccbb1eb248"} Oct 30 13:27:01.091541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.091430 5693 scope.go:122] "RemoveContainer" containerID="fb5e00cfbc9828b2d137ba8906106bb9e907a860992c60f02ddafbc9738bf719" Oct 30 13:27:01.092930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.092906 5693 generic.go:356] "Generic (PLEG): container finished" podID="74bb88af-59ff-426d-b070-c9a4e8944e88" containerID="eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3" exitCode=0 Oct 30 13:27:01.093044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.092947 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" Oct 30 13:27:01.093044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.092942 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" event={"ID":"74bb88af-59ff-426d-b070-c9a4e8944e88","Type":"ContainerDied","Data":"eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3"} Oct 30 13:27:01.093154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.093057 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp" event={"ID":"74bb88af-59ff-426d-b070-c9a4e8944e88","Type":"ContainerDied","Data":"4b5080bca166052bcaf0dc50a3580159c04ae1d2d60626a00d55c57378a7a474"} Oct 30 13:27:01.093203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.093150 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm" Oct 30 13:27:01.107926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.107903 5693 scope.go:122] "RemoveContainer" containerID="eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3" Oct 30 13:27:01.119946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.119917 5693 scope.go:122] "RemoveContainer" containerID="eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3" Oct 30 13:27:01.120248 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:01.120226 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3\": container with ID starting with eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3 not found: ID does not exist" containerID="eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3" Oct 30 13:27:01.120332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.120257 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3"} err="failed to get container status \"eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3\": rpc error: code = NotFound desc = could not find container \"eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3\": container with ID starting with eaa8be08e9b758413322fd1b50e7da93fc695fc1a140ece163bb705d8c54c6c3 not found: ID does not exist" Oct 30 13:27:01.145746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.145718 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm"] Oct 30 13:27:01.151282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.151258 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1f486a-79bf5bbcf-gs7qm"] Oct 30 13:27:01.206456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.206425 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp"] Oct 30 13:27:01.209358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.209331 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-7989cf44cf-wk8rp"] Oct 30 13:27:01.240552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.240517 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk"] Oct 30 13:27:01.245831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.245809 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-6cff6f6c48-cbbvk"] Oct 30 13:27:01.293009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:01.292980 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:02.254989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:02.254955 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439e921f-9e85-4bcd-a458-a7de27e0ca70" path="/var/lib/kubelet/pods/439e921f-9e85-4bcd-a458-a7de27e0ca70/volumes" Oct 30 13:27:02.255415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:02.255400 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bb88af-59ff-426d-b070-c9a4e8944e88" path="/var/lib/kubelet/pods/74bb88af-59ff-426d-b070-c9a4e8944e88/volumes" Oct 30 13:27:02.255831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:02.255814 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96da0066-a10e-4b12-9284-8635da91d592" path="/var/lib/kubelet/pods/96da0066-a10e-4b12-9284-8635da91d592/volumes" Oct 30 13:27:02.917963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:02.917771 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:04.700620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:04.699239 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:07.056621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:07.055686 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:07.082905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:07.070750 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:07.275620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:07.273113 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:07.312670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:07.303933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:09.370636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:09.366284 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:09.616729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:09.616698 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c"] Oct 30 13:27:09.617183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:09.617131 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" containerID="cri-o://60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7" gracePeriod=30 Oct 30 13:27:09.789719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:09.735337 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:09.959688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:09.951457 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:10.462623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.462193 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf"] Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463267 5693 cpu_manager.go:414] "containerMap: removing container" podUID="439e921f-9e85-4bcd-a458-a7de27e0ca70" containerName="payload-processing" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463291 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439e921f-9e85-4bcd-a458-a7de27e0ca70" containerName="payload-processing" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463352 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74bb88af-59ff-426d-b070-c9a4e8944e88" containerName="payload-pre-processing" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463361 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bb88af-59ff-426d-b070-c9a4e8944e88" containerName="payload-pre-processing" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463386 5693 cpu_manager.go:414] "containerMap: removing container" podUID="96da0066-a10e-4b12-9284-8635da91d592" containerName="maas-api" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463395 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96da0066-a10e-4b12-9284-8635da91d592" containerName="maas-api" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463557 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bb88af-59ff-426d-b070-c9a4e8944e88" containerName="payload-pre-processing" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463574 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="96da0066-a10e-4b12-9284-8635da91d592" containerName="maas-api" Oct 30 13:27:10.466620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.463592 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="439e921f-9e85-4bcd-a458-a7de27e0ca70" containerName="payload-processing" Oct 30 13:27:10.479621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.469190 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.487195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.487160 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf"] Oct 30 13:27:10.503919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.503885 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504015 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504060 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504140 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffbh\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504225 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504291 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504347 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504373 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.504412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.504401 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.605859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.605817 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.605899 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.605923 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.605947 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffbh\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.605982 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606036 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606086 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606114 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606143 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606175 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606487 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606764 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.606892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606866 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.607008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.606992 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.608938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.608916 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.610186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.610158 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.671622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.671278 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.675620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.673229 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffbh\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.794654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.794624 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:10.984646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.980176 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:27:10.984646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:10.980248 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:27:11.184288 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:27:11.184259 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239559a4_f5c0_4b59_85d3_3cd052a548de.slice/crio-3ed89f77f4d3ec0e7ad7d6b8b6a2fd7980c6495c39bfd391145391176ba4b721 WatchSource:0}: Error finding container 3ed89f77f4d3ec0e7ad7d6b8b6a2fd7980c6495c39bfd391145391176ba4b721: Status 404 returned error can't find the container with id 3ed89f77f4d3ec0e7ad7d6b8b6a2fd7980c6495c39bfd391145391176ba4b721 Oct 30 13:27:11.184672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.184650 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf"] Oct 30 13:27:11.186883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.186851 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:11.186956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.186918 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:11.186956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.186948 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:11.609724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.609515 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" event={"ID":"239559a4-f5c0-4b59-85d3-3cd052a548de","Type":"ContainerStarted","Data":"af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475"} Oct 30 13:27:11.609724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.609576 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" event={"ID":"239559a4-f5c0-4b59-85d3-3cd052a548de","Type":"ContainerStarted","Data":"3ed89f77f4d3ec0e7ad7d6b8b6a2fd7980c6495c39bfd391145391176ba4b721"} Oct 30 13:27:11.663831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.663770 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podStartSLOduration=1.6637487499999999 podStartE2EDuration="1.66374875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:11.658064596 +0000 UTC m=+1407.524682770" watchObservedRunningTime="2026-10-30 13:27:11.66374875 +0000 UTC m=+1407.530366924" Oct 30 13:27:11.795905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:11.795864 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:12.304679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.304568 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:12.368329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.368296 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28"] Oct 30 13:27:12.369361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.369331 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" containerID="cri-o://7b43a6ee0702ba0afa8a310fe8188a29932686b0f5167bcfa00056dbb75a450b" gracePeriod=30 Oct 30 13:27:12.398711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.398673 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w"] Oct 30 13:27:12.399162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.399107 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" containerID="cri-o://9afe48ff829b9833582f429c6c5601a5094df481aa69ff516f581fd1c8f62859" gracePeriod=30 Oct 30 13:27:12.401858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.401830 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4"] Oct 30 13:27:12.402047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.402025 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" containerID="cri-o://702ae4e3e2878c2ba1ec7e4b64a2a5d347b1453b21fb85c162333e35aad1456c" gracePeriod=30 Oct 30 13:27:12.797342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.797310 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:12.797721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:12.797377 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:13.181019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:13.180889 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.398727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:13.398692 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk"] Oct 30 13:27:13.566633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:13.566575 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.792004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:13.791970 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.795574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:13.795267 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:13.795574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:13.795316 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:14.423077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:14.423042 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:14.795739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:14.795697 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:14.795943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:14.795762 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:15.011692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.011660 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:27:15.155084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.154991 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155119 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155170 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155206 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155264 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155401 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155566 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155618 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.155725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.155670 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo\") pod \"129df7be-7639-4b69-a599-2f45f8670f08\" (UID: \"129df7be-7639-4b69-a599-2f45f8670f08\") " Oct 30 13:27:15.157903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.157688 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:15.158429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.158120 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:15.158429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.158343 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:15.158429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.158405 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:15.159451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.158714 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:15.166008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.165886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:15.166008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.165961 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:15.166308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.166274 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:15.168166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.168129 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr" pod "129df7be-7639-4b69-a599-2f45f8670f08" (UID: "129df7be-7639-4b69-a599-2f45f8670f08"). InnerVolumeSpecName "kube-api-access-cwnxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:15.257508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257462 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257508 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257520 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257532 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257543 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/129df7be-7639-4b69-a599-2f45f8670f08-kube-api-access-cwnxr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257556 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129df7be-7639-4b69-a599-2f45f8670f08-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257567 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129df7be-7639-4b69-a599-2f45f8670f08-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257578 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.257751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.257589 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129df7be-7639-4b69-a599-2f45f8670f08-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:15.754832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.754788 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:27:15.755455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.755417 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:15.771581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.771545 5693 generic.go:356] "Generic (PLEG): container finished" podID="129df7be-7639-4b69-a599-2f45f8670f08" containerID="60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7" exitCode=0 Oct 30 13:27:15.771756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.771616 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" Oct 30 13:27:15.771756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.771628 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" event={"ID":"129df7be-7639-4b69-a599-2f45f8670f08","Type":"ContainerDied","Data":"60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7"} Oct 30 13:27:15.771756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.771694 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c" event={"ID":"129df7be-7639-4b69-a599-2f45f8670f08","Type":"ContainerDied","Data":"aa95bba61ed636a95d7230ba500aaf218a5ea2fb5a90c145f75fdb8f57b98036"} Oct 30 13:27:15.771756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.771727 5693 scope.go:122] "RemoveContainer" containerID="60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7" Oct 30 13:27:15.785042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.785015 5693 scope.go:122] "RemoveContainer" containerID="ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750" Oct 30 13:27:15.795319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.795280 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:15.795549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.795512 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:15.801006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.797747 5693 scope.go:122] "RemoveContainer" containerID="60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7" Oct 30 13:27:15.801006 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:15.798093 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7\": container with ID starting with 60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7 not found: ID does not exist" containerID="60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7" Oct 30 13:27:15.801006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.798128 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7"} err="failed to get container status \"60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7\": rpc error: code = NotFound desc = could not find container \"60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7\": container with ID starting with 60675d5bb93ef353d168ebd108656f90eb5f80a6ab5da1a1b6b4329cec56f2b7 not found: ID does not exist" Oct 30 13:27:15.801006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.798150 5693 scope.go:122] "RemoveContainer" containerID="ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750" Oct 30 13:27:15.801006 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:15.798559 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750\": container with ID starting with ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750 not found: ID does not exist" containerID="ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750" Oct 30 13:27:15.801006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.798589 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750"} err="failed to get container status \"ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750\": rpc error: code = NotFound desc = could not find container \"ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750\": container with ID starting with ebb190eea05538084dddb174cae1e0046fc485a2f6450da43a3a52a2576c8750 not found: ID does not exist" Oct 30 13:27:15.856685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.854647 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c"] Oct 30 13:27:15.889635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:15.888313 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1f486a-openshift-default-76dc864d4f-jgl4c"] Oct 30 13:27:16.266965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:16.266919 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129df7be-7639-4b69-a599-2f45f8670f08" path="/var/lib/kubelet/pods/129df7be-7639-4b69-a599-2f45f8670f08/volumes" Oct 30 13:27:16.390107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:16.389883 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:16.795376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:16.795344 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:16.795894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:16.795409 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:17.795648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.795611 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:17.796558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.796525 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:17.800529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.798839 5693 generic.go:356] "Generic (PLEG): container finished" podID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerID="702ae4e3e2878c2ba1ec7e4b64a2a5d347b1453b21fb85c162333e35aad1456c" exitCode=0 Oct 30 13:27:17.800529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.799076 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" event={"ID":"40b1c585-2d9d-498a-b1ed-d80e5ad5e432","Type":"ContainerDied","Data":"702ae4e3e2878c2ba1ec7e4b64a2a5d347b1453b21fb85c162333e35aad1456c"} Oct 30 13:27:17.802314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.802287 5693 generic.go:356] "Generic (PLEG): container finished" podID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerID="9afe48ff829b9833582f429c6c5601a5094df481aa69ff516f581fd1c8f62859" exitCode=0 Oct 30 13:27:17.802429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.802408 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" event={"ID":"12e887d1-d90f-4a31-b99e-a72e3864802d","Type":"ContainerDied","Data":"9afe48ff829b9833582f429c6c5601a5094df481aa69ff516f581fd1c8f62859"} Oct 30 13:27:17.804248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.804223 5693 generic.go:356] "Generic (PLEG): container finished" podID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerID="7b43a6ee0702ba0afa8a310fe8188a29932686b0f5167bcfa00056dbb75a450b" exitCode=0 Oct 30 13:27:17.804340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.804271 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" event={"ID":"f7eda9b4-c641-4abf-8db7-41086ce4b90f","Type":"ContainerDied","Data":"7b43a6ee0702ba0afa8a310fe8188a29932686b0f5167bcfa00056dbb75a450b"} Oct 30 13:27:17.804340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.804312 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" event={"ID":"f7eda9b4-c641-4abf-8db7-41086ce4b90f","Type":"ContainerDied","Data":"89af882d93b5a1bfbe23b9242118ecc383c724080f8e739a29d414b17ff16383"} Oct 30 13:27:17.804449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.804357 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89af882d93b5a1bfbe23b9242118ecc383c724080f8e739a29d414b17ff16383" Oct 30 13:27:17.812878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.812852 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.868742 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.868842 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.868874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.868916 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.868946 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.868967 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.869007 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.869029 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.869062 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket\") pod \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\" (UID: \"f7eda9b4-c641-4abf-8db7-41086ce4b90f\") " Oct 30 13:27:17.869822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.869769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.870841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.870092 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:17.870841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.870232 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:27:17.870841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.870326 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.870841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.870363 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.870841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.870396 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.873714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.873686 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:17.873828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.873798 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.874514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.874478 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "kube-api-access-7pmt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:17.876680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.875898 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo" pod "f7eda9b4-c641-4abf-8db7-41086ce4b90f" (UID: "f7eda9b4-c641-4abf-8db7-41086ce4b90f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:17.878822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.878724 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:27:17.970356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970264 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970564 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970618 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970731 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970784 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970818 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.970883 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971037 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971086 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971100 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971126 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.970652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971185 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.971918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971250 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.971918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971321 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.971918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971370 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.971918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971420 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.971918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.971483 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.972259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.972173 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.972386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.972360 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.972982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.972943 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:17.972982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.972968 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.973269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.973047 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.973269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.973075 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.973269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.973105 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.973269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.973182 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data\") pod \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\" (UID: \"40b1c585-2d9d-498a-b1ed-d80e5ad5e432\") " Oct 30 13:27:17.973269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.973223 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z\") pod \"12e887d1-d90f-4a31-b99e-a72e3864802d\" (UID: \"12e887d1-d90f-4a31-b99e-a72e3864802d\") " Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.973741 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974085 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974106 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974119 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974132 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974145 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974159 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974173 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974187 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974201 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974215 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974228 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974242 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974256 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974259 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974269 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmt9\" (UniqueName: \"kubernetes.io/projected/f7eda9b4-c641-4abf-8db7-41086ce4b90f-kube-api-access-7pmt9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974283 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974297 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974311 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.974346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.974325 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eda9b4-c641-4abf-8db7-41086ce4b90f-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.981297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.981247 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:17.989052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.988906 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.989052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.989041 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "kube-api-access-5tdnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:17.989219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.989144 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:17.989997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.989963 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:17.990171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.990077 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo" pod "40b1c585-2d9d-498a-b1ed-d80e5ad5e432" (UID: "40b1c585-2d9d-498a-b1ed-d80e5ad5e432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:17.992431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.992396 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:17.995561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:17.995534 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z" pod "12e887d1-d90f-4a31-b99e-a72e3864802d" (UID: "12e887d1-d90f-4a31-b99e-a72e3864802d"). InnerVolumeSpecName "kube-api-access-wh98z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076010 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh98z\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-kube-api-access-wh98z\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076057 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076073 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076086 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e887d1-d90f-4a31-b99e-a72e3864802d-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076099 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076115 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076133 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdnm\" (UniqueName: \"kubernetes.io/projected/40b1c585-2d9d-498a-b1ed-d80e5ad5e432-kube-api-access-5tdnm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076151 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.076248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.076167 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e887d1-d90f-4a31-b99e-a72e3864802d-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.399072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.399022 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" podUID="c5dfc294-6cac-4419-82b3-93a5b2f11a19" containerName="payload-pre-processing" containerID="cri-o://36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264" gracePeriod=25 Oct 30 13:27:18.466435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.466184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:18.722569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.722540 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:27:18.799053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.798970 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg\") pod \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " Oct 30 13:27:18.799514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.799203 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume\") pod \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\" (UID: \"c5dfc294-6cac-4419-82b3-93a5b2f11a19\") " Oct 30 13:27:18.800385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.800355 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume" pod "c5dfc294-6cac-4419-82b3-93a5b2f11a19" (UID: "c5dfc294-6cac-4419-82b3-93a5b2f11a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:18.803469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.803420 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg" pod "c5dfc294-6cac-4419-82b3-93a5b2f11a19" (UID: "c5dfc294-6cac-4419-82b3-93a5b2f11a19"). InnerVolumeSpecName "kube-api-access-tktbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:18.803583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.803554 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:27:18.803671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.803646 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:27:18.816409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.816375 5693 generic.go:356] "Generic (PLEG): container finished" podID="4e947291-0a68-48df-b65a-d3eeda038447" containerID="ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9" exitCode=0 Oct 30 13:27:18.816590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.816519 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" event={"ID":"4e947291-0a68-48df-b65a-d3eeda038447","Type":"ContainerDied","Data":"ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9"} Oct 30 13:27:18.817089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.817067 5693 scope.go:122] "RemoveContainer" containerID="ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9" Oct 30 13:27:18.820408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.820382 5693 generic.go:356] "Generic (PLEG): container finished" podID="c5dfc294-6cac-4419-82b3-93a5b2f11a19" containerID="36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264" exitCode=0 Oct 30 13:27:18.820514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.820445 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" event={"ID":"c5dfc294-6cac-4419-82b3-93a5b2f11a19","Type":"ContainerDied","Data":"36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264"} Oct 30 13:27:18.820514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.820490 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" event={"ID":"c5dfc294-6cac-4419-82b3-93a5b2f11a19","Type":"ContainerDied","Data":"4d0070d695155cd6be6666b23192b7c00d173fbbb6fab9ffba074368ca7b7182"} Oct 30 13:27:18.820643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.820522 5693 scope.go:122] "RemoveContainer" containerID="36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264" Oct 30 13:27:18.820643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.820631 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk" Oct 30 13:27:18.824625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.823925 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:18.824625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.824012 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:18.824625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.824060 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:18.830239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.825877 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" event={"ID":"12e887d1-d90f-4a31-b99e-a72e3864802d","Type":"ContainerDied","Data":"ff32b9edda46019030a1369a820aa19454c6e6475f69ff621fe21c9303065ded"} Oct 30 13:27:18.830239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.825989 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w" Oct 30 13:27:18.830239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.828428 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28" Oct 30 13:27:18.830239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.829560 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" Oct 30 13:27:18.830239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.829658 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4" event={"ID":"40b1c585-2d9d-498a-b1ed-d80e5ad5e432","Type":"ContainerDied","Data":"e387db2331b1b2e893e8767eaed3077cfd2dadbe5a4c5bba44de79b66de1ecd3"} Oct 30 13:27:18.862214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.862190 5693 scope.go:122] "RemoveContainer" containerID="36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264" Oct 30 13:27:18.863989 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:18.863956 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264\": container with ID starting with 36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264 not found: ID does not exist" containerID="36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264" Oct 30 13:27:18.864100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.864001 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264"} err="failed to get container status \"36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264\": rpc error: code = NotFound desc = could not find container \"36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264\": container with ID starting with 36c85b115b1e8c62ff2378c5455295beb922ec76c21b5e8692334b609df8b264 not found: ID does not exist" Oct 30 13:27:18.864100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.864024 5693 scope.go:122] "RemoveContainer" containerID="9afe48ff829b9833582f429c6c5601a5094df481aa69ff516f581fd1c8f62859" Oct 30 13:27:18.902679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.902646 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/c5dfc294-6cac-4419-82b3-93a5b2f11a19-kube-api-access-tktbg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.902679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.902682 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dfc294-6cac-4419-82b3-93a5b2f11a19-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.915734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.915706 5693 scope.go:122] "RemoveContainer" containerID="702ae4e3e2878c2ba1ec7e4b64a2a5d347b1453b21fb85c162333e35aad1456c" Oct 30 13:27:18.970230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.970199 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4"] Oct 30 13:27:18.984192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:18.984161 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zklq4"] Oct 30 13:27:19.019158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.019097 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28"] Oct 30 13:27:19.023978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.023949 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fkt28"] Oct 30 13:27:19.063658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.058217 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w"] Oct 30 13:27:19.066184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.066157 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k7v2w"] Oct 30 13:27:19.110087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.110003 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk"] Oct 30 13:27:19.146025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.142311 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-bd788f9cf-rvcmk"] Oct 30 13:27:19.869342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.869304 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" event={"ID":"4e947291-0a68-48df-b65a-d3eeda038447","Type":"ContainerStarted","Data":"b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25"} Oct 30 13:27:19.871069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.871045 5693 generic.go:356] "Generic (PLEG): container finished" podID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerID="e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6" exitCode=0 Oct 30 13:27:19.871162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.871139 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" event={"ID":"7b065a33-cc37-42bd-b49b-cb1bff218923","Type":"ContainerDied","Data":"e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6"} Oct 30 13:27:19.871484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.871467 5693 scope.go:122] "RemoveContainer" containerID="e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6" Oct 30 13:27:19.873062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.873029 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:19.873166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.873108 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:19.873166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.873150 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:19.894937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.894486 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:19.894937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.894803 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:19.895904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:19.895881 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:27:20.270744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.270703 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" path="/var/lib/kubelet/pods/12e887d1-d90f-4a31-b99e-a72e3864802d/volumes" Oct 30 13:27:20.271518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.271487 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" path="/var/lib/kubelet/pods/40b1c585-2d9d-498a-b1ed-d80e5ad5e432/volumes" Oct 30 13:27:20.272162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.272135 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5dfc294-6cac-4419-82b3-93a5b2f11a19" path="/var/lib/kubelet/pods/c5dfc294-6cac-4419-82b3-93a5b2f11a19/volumes" Oct 30 13:27:20.273262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.273239 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" path="/var/lib/kubelet/pods/f7eda9b4-c641-4abf-8db7-41086ce4b90f/volumes" Oct 30 13:27:20.400200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.399164 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:20.723493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.723395 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:27:20.723730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.723633 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:27:20.723730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.723689 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:27:20.887044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:20.887007 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" event={"ID":"7b065a33-cc37-42bd-b49b-cb1bff218923","Type":"ContainerStarted","Data":"3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309"} Oct 30 13:27:21.078688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:21.077385 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:27:21.078688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:21.078173 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:27:21.078688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:21.078224 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 13:27:21.098520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:21.098483 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:21.723141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:21.723108 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:27:21.723302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:21.723161 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:27:22.077841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:22.075937 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:27:22.077841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:22.075994 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 13:27:22.440676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:22.435494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:22.724034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:22.723997 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:27:22.724219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:22.724063 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:27:23.084501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:23.083789 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:27:23.084501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:23.083851 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 13:27:23.109873 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:23.109838 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:27:23.723180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:23.723143 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:27:23.723359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:23.723197 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:27:24.081284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:24.079926 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:27:24.081284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:24.079987 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 13:27:24.495619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:24.494387 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:24.724079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:24.724048 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:27:24.724316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:24.724294 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:27:25.075584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:25.075551 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:27:25.075809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:25.075627 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 13:27:25.602072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:25.602038 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:27:25.739482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:25.737876 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:27:25.739482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:25.739322 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:27:25.748511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:25.748450 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:27:26.079621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:26.079219 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:27:26.079621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:26.079278 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 13:27:26.425825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:26.425743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:27.079258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:27.079229 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:27:27.080323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:27.080301 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:27:28.167111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:28.167076 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:30.600755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:30.600639 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:32.113938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.113316 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:32.325125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.325091 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:32.553799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.553762 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb"] Oct 30 13:27:32.554452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554435 5693 cpu_manager.go:414] "containerMap: removing container" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" Oct 30 13:27:32.554452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554452 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554473 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c5dfc294-6cac-4419-82b3-93a5b2f11a19" containerName="payload-pre-processing" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554479 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dfc294-6cac-4419-82b3-93a5b2f11a19" containerName="payload-pre-processing" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554504 5693 cpu_manager.go:414] "containerMap: removing container" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554510 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554523 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554528 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554540 5693 cpu_manager.go:414] "containerMap: removing container" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" Oct 30 13:27:32.554590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554546 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" Oct 30 13:27:32.554873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554709 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" Oct 30 13:27:32.554873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554727 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" Oct 30 13:27:32.554873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554741 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dfc294-6cac-4419-82b3-93a5b2f11a19" containerName="payload-pre-processing" Oct 30 13:27:32.554873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554751 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e887d1-d90f-4a31-b99e-a72e3864802d" containerName="istio-proxy" Oct 30 13:27:32.554873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554759 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eda9b4-c641-4abf-8db7-41086ce4b90f" containerName="istio-proxy" Oct 30 13:27:32.554873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.554766 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b1c585-2d9d-498a-b1ed-d80e5ad5e432" containerName="istio-proxy" Oct 30 13:27:32.559698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.559658 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.580006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.579977 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb"] Oct 30 13:27:32.588273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.588243 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-65e250-dockercfg-9bllj\"" Oct 30 13:27:32.588444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.588402 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-65e250\"" Oct 30 13:27:32.597221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.597188 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxg4\" (UniqueName: \"kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4\") pod \"payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.597365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.597313 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.698266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.698230 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxg4\" (UniqueName: \"kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4\") pod \"payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.698441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.698331 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.699050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.699028 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.721225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.721189 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxg4\" (UniqueName: \"kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4\") pod \"payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.726863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.726839 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq"] Oct 30 13:27:32.727557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.727541 5693 cpu_manager.go:414] "containerMap: removing container" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" Oct 30 13:27:32.727632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.727560 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129df7be-7639-4b69-a599-2f45f8670f08" containerName="istio-proxy" Oct 30 13:27:32.731409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.731391 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:32.754468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.754440 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq"] Oct 30 13:27:32.799732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.799695 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j9sm\" (UniqueName: \"kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm\") pod \"payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:32.799732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.799733 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:32.874703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.874585 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:32.901381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.901343 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j9sm\" (UniqueName: \"kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm\") pod \"payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:32.901536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.901398 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:32.902154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.902135 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:32.922291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:32.922262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j9sm\" (UniqueName: \"kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm\") pod \"payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:33.050032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:33.049996 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:33.202769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:33.202740 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb"] Oct 30 13:27:33.374719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:33.374692 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq"] Oct 30 13:27:33.377349 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:27:33.377322 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e797b2_9872_4e07_9995_395ff263cec3.slice/crio-42bd23990cde6e31f079a93bdada748d0b16e9cb66f0a27780cb4106b475e670 WatchSource:0}: Error finding container 42bd23990cde6e31f079a93bdada748d0b16e9cb66f0a27780cb4106b475e670: Status 404 returned error can't find the container with id 42bd23990cde6e31f079a93bdada748d0b16e9cb66f0a27780cb4106b475e670 Oct 30 13:27:33.478530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:33.478376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:33.753010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:33.752977 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc"] Oct 30 13:27:33.784848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:33.784814 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67"] Oct 30 13:27:34.040691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.039514 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" event={"ID":"f20c80d5-5898-4508-9de3-d1d179dac629","Type":"ContainerStarted","Data":"cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e"} Oct 30 13:27:34.040691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.039765 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" event={"ID":"f20c80d5-5898-4508-9de3-d1d179dac629","Type":"ContainerStarted","Data":"75b5faacbc7d030c2b4df678e8ddbb96ed159d9e98e28cde3e9450ec723fad40"} Oct 30 13:27:34.040949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.040809 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:34.045157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.044980 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" event={"ID":"95e797b2-9872-4e07-9995-395ff263cec3","Type":"ContainerStarted","Data":"8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916"} Oct 30 13:27:34.045157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.045027 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" event={"ID":"95e797b2-9872-4e07-9995-395ff263cec3","Type":"ContainerStarted","Data":"42bd23990cde6e31f079a93bdada748d0b16e9cb66f0a27780cb4106b475e670"} Oct 30 13:27:34.046059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.045431 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:34.094227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.094178 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" podStartSLOduration=2.094164376 podStartE2EDuration="2.094164376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:34.093801185 +0000 UTC m=+1429.960419358" watchObservedRunningTime="2026-10-30 13:27:34.094164376 +0000 UTC m=+1429.960782549" Oct 30 13:27:34.149340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:34.149290 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" podStartSLOduration=2.149272522 podStartE2EDuration="2.149272522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:34.147942989 +0000 UTC m=+1430.014561163" watchObservedRunningTime="2026-10-30 13:27:34.149272522 +0000 UTC m=+1430.015890689" Oct 30 13:27:35.203770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:35.203739 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:35.704863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:35.704831 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:37.606621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:37.596284 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:38.753849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:38.753808 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" podUID="cc06a9a5-c433-4023-95e7-861839c2266f" containerName="payload-processing" containerID="cri-o://8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e" gracePeriod=25 Oct 30 13:27:38.785086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:38.785032 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" podUID="18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" containerName="payload-pre-processing" containerID="cri-o://4132a037b648ccf393f6beb880e82d2b27e360cc13d8ddd71b1ca252f535f4e4" gracePeriod=25 Oct 30 13:27:39.073935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.073905 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:27:39.086648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.086613 5693 generic.go:356] "Generic (PLEG): container finished" podID="18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" containerID="4132a037b648ccf393f6beb880e82d2b27e360cc13d8ddd71b1ca252f535f4e4" exitCode=0 Oct 30 13:27:39.086807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.086651 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" event={"ID":"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0","Type":"ContainerDied","Data":"4132a037b648ccf393f6beb880e82d2b27e360cc13d8ddd71b1ca252f535f4e4"} Oct 30 13:27:39.088207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.088187 5693 generic.go:356] "Generic (PLEG): container finished" podID="cc06a9a5-c433-4023-95e7-861839c2266f" containerID="8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e" exitCode=0 Oct 30 13:27:39.088301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.088240 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" Oct 30 13:27:39.088301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.088273 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" event={"ID":"cc06a9a5-c433-4023-95e7-861839c2266f","Type":"ContainerDied","Data":"8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e"} Oct 30 13:27:39.088380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.088323 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc" event={"ID":"cc06a9a5-c433-4023-95e7-861839c2266f","Type":"ContainerDied","Data":"67f1e70af0c7072265b92317b198f6e9723d0cf3a4aac56f2fd4df98ee9af82d"} Oct 30 13:27:39.088380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.088348 5693 scope.go:122] "RemoveContainer" containerID="8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e" Oct 30 13:27:39.108015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.107999 5693 scope.go:122] "RemoveContainer" containerID="8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e" Oct 30 13:27:39.108264 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:39.108246 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e\": container with ID starting with 8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e not found: ID does not exist" containerID="8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e" Oct 30 13:27:39.108318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.108274 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e"} err="failed to get container status \"8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e\": rpc error: code = NotFound desc = could not find container \"8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e\": container with ID starting with 8e6443f75dec10a9dc6db93f6220bbe3e32fb1c88684f4cec85975f8845f206e not found: ID does not exist" Oct 30 13:27:39.110027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.110012 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:27:39.222401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.222371 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume\") pod \"cc06a9a5-c433-4023-95e7-861839c2266f\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " Oct 30 13:27:39.222584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.222480 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume\") pod \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " Oct 30 13:27:39.222648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.222617 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9\" (UniqueName: \"kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9\") pod \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\" (UID: \"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0\") " Oct 30 13:27:39.222688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.222677 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7\" (UniqueName: \"kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7\") pod \"cc06a9a5-c433-4023-95e7-861839c2266f\" (UID: \"cc06a9a5-c433-4023-95e7-861839c2266f\") " Oct 30 13:27:39.222858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.222838 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume" pod "cc06a9a5-c433-4023-95e7-861839c2266f" (UID: "cc06a9a5-c433-4023-95e7-861839c2266f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:39.222905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.222853 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume" pod "18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" (UID: "18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:39.223220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.223198 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06a9a5-c433-4023-95e7-861839c2266f-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:39.223220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.223218 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:39.224821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.224797 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7" pod "cc06a9a5-c433-4023-95e7-861839c2266f" (UID: "cc06a9a5-c433-4023-95e7-861839c2266f"). InnerVolumeSpecName "kube-api-access-27ds7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:39.224927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.224828 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9" pod "18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" (UID: "18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0"). InnerVolumeSpecName "kube-api-access-g5xg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:39.324851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.324772 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xg9\" (UniqueName: \"kubernetes.io/projected/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0-kube-api-access-g5xg9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:39.324851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.324799 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27ds7\" (UniqueName: \"kubernetes.io/projected/cc06a9a5-c433-4023-95e7-861839c2266f-kube-api-access-27ds7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:39.353627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.353524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:39.438311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.438274 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc"] Oct 30 13:27:39.444155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:39.444127 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d797fc-78d94755f6-fddjc"] Oct 30 13:27:40.097223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.097140 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" event={"ID":"18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0","Type":"ContainerDied","Data":"d01489479e59081a8fa48de2f277777325047c5a1132a09847c0f53586790e7e"} Oct 30 13:27:40.097223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.097173 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67" Oct 30 13:27:40.097223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.097202 5693 scope.go:122] "RemoveContainer" containerID="4132a037b648ccf393f6beb880e82d2b27e360cc13d8ddd71b1ca252f535f4e4" Oct 30 13:27:40.155335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.155298 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67"] Oct 30 13:27:40.161502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.161475 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d797fc-86949fd695-tpp67"] Oct 30 13:27:40.254964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.254929 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" path="/var/lib/kubelet/pods/18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0/volumes" Oct 30 13:27:40.255428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:40.255408 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc06a9a5-c433-4023-95e7-861839c2266f" path="/var/lib/kubelet/pods/cc06a9a5-c433-4023-95e7-861839c2266f/volumes" Oct 30 13:27:41.094277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:41.094238 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:42.756626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:42.727694 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:43.614343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:43.614311 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn"] Oct 30 13:27:43.643677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:43.643644 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj"] Oct 30 13:27:43.812093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:43.812034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:43.981104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:43.980955 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:44.498551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:44.498514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:45.892139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:45.892104 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5"] Oct 30 13:27:45.892631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:45.892358 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" podUID="1ea6e696-7f58-4427-b35b-b59014a4bd9a" containerName="maas-api" containerID="cri-o://f98c287b4ca54380ba8ff871532d333b03217ecfbb6ebf2e4029023aeef1deeb" gracePeriod=30 Oct 30 13:27:46.008693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.008664 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:46.063150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.063121 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:27:46.063453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.063431 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:27:46.145065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.145034 5693 generic.go:356] "Generic (PLEG): container finished" podID="1ea6e696-7f58-4427-b35b-b59014a4bd9a" containerID="f98c287b4ca54380ba8ff871532d333b03217ecfbb6ebf2e4029023aeef1deeb" exitCode=0 Oct 30 13:27:46.145226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.145130 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" event={"ID":"1ea6e696-7f58-4427-b35b-b59014a4bd9a","Type":"ContainerDied","Data":"f98c287b4ca54380ba8ff871532d333b03217ecfbb6ebf2e4029023aeef1deeb"} Oct 30 13:27:46.145226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.145191 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" event={"ID":"1ea6e696-7f58-4427-b35b-b59014a4bd9a","Type":"ContainerDied","Data":"a1163647796e6d4730307646a22669823e0c22b635d1df932c18ba088208dfe2"} Oct 30 13:27:46.145226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.145214 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1163647796e6d4730307646a22669823e0c22b635d1df932c18ba088208dfe2" Oct 30 13:27:46.168962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.168926 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:27:46.244438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.244409 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\") pod \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " Oct 30 13:27:46.244724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.244706 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5\" (UniqueName: \"kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5\") pod \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " Oct 30 13:27:46.244830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.244745 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs\") pod \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " Oct 30 13:27:46.244954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.244933 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca\") pod \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\" (UID: \"1ea6e696-7f58-4427-b35b-b59014a4bd9a\") " Oct 30 13:27:46.245490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.245463 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca" pod "1ea6e696-7f58-4427-b35b-b59014a4bd9a" (UID: "1ea6e696-7f58-4427-b35b-b59014a4bd9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:46.245832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.245812 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea6e696-7f58-4427-b35b-b59014a4bd9a-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.247591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.247560 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5" pod "1ea6e696-7f58-4427-b35b-b59014a4bd9a" (UID: "1ea6e696-7f58-4427-b35b-b59014a4bd9a"). InnerVolumeSpecName "kube-api-access-7vbq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:46.247789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.247585 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls" pod "1ea6e696-7f58-4427-b35b-b59014a4bd9a" (UID: "1ea6e696-7f58-4427-b35b-b59014a4bd9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:46.247859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.247800 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs" pod "1ea6e696-7f58-4427-b35b-b59014a4bd9a" (UID: "1ea6e696-7f58-4427-b35b-b59014a4bd9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:46.356624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.349793 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vbq5\" (UniqueName: \"kubernetes.io/projected/1ea6e696-7f58-4427-b35b-b59014a4bd9a-kube-api-access-7vbq5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.356624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.349829 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.356624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.349842 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea6e696-7f58-4427-b35b-b59014a4bd9a-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.588491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.588454 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:46.792772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:46.789101 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:47.152336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:47.152304 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5" Oct 30 13:27:47.212923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:47.212888 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5"] Oct 30 13:27:47.223309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:47.223278 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d797fc-5f4446c88b-k2wx5"] Oct 30 13:27:47.285844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:47.285818 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:47.303287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:47.303261 5693 scope.go:122] "RemoveContainer" containerID="99ee0112a9b9ff0e8fc20f194ff1f0946fde1d9c5b1a34c6c85891c86d30aeee" Oct 30 13:27:47.313012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:47.312989 5693 scope.go:122] "RemoveContainer" containerID="7b43a6ee0702ba0afa8a310fe8188a29932686b0f5167bcfa00056dbb75a450b" Oct 30 13:27:48.255342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:48.255300 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea6e696-7f58-4427-b35b-b59014a4bd9a" path="/var/lib/kubelet/pods/1ea6e696-7f58-4427-b35b-b59014a4bd9a/volumes" Oct 30 13:27:48.615309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:48.615179 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerName="payload-processing" containerID="cri-o://ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd" gracePeriod=25 Oct 30 13:27:48.644913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:48.644866 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerName="payload-pre-processing" containerID="cri-o://9f0d325304a2cc2c716b0a666f9613493f3ad14e90f6db9fe5ce745e122294fd" gracePeriod=25 Oct 30 13:27:48.836747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:48.836713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.173982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.173950 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.626392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.626359 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz"] Oct 30 13:27:50.627087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627065 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cc06a9a5-c433-4023-95e7-861839c2266f" containerName="payload-processing" Oct 30 13:27:50.627193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627087 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc06a9a5-c433-4023-95e7-861839c2266f" containerName="payload-processing" Oct 30 13:27:50.627193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627107 5693 cpu_manager.go:414] "containerMap: removing container" podUID="18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" containerName="payload-pre-processing" Oct 30 13:27:50.627193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627114 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" containerName="payload-pre-processing" Oct 30 13:27:50.627357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627194 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1ea6e696-7f58-4427-b35b-b59014a4bd9a" containerName="maas-api" Oct 30 13:27:50.627357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627202 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea6e696-7f58-4427-b35b-b59014a4bd9a" containerName="maas-api" Oct 30 13:27:50.627464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627356 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea6e696-7f58-4427-b35b-b59014a4bd9a" containerName="maas-api" Oct 30 13:27:50.627464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627371 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc06a9a5-c433-4023-95e7-861839c2266f" containerName="payload-processing" Oct 30 13:27:50.627464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.627390 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ff25d3-ae74-4dc3-aee8-9f5ab7def0b0" containerName="payload-pre-processing" Oct 30 13:27:50.631026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.631005 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.662553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.662518 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-642765-openshift-default-dockercfg-gjqhm\"" Oct 30 13:27:50.666914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.666884 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz"] Oct 30 13:27:50.710935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.710896 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711030 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711096 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711148 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711168 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qxl\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711185 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711389 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711516 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.711671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.711650 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.813849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.813725 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.817655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.814187 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.824881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.817896 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.825353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.825323 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.825531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.825515 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qxl\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.825682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.825665 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.826098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.826072 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.838673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.826460 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.839016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.838989 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.839655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.839631 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.839952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.839924 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.840771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.840749 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.840899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.840753 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.844727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.841463 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.854178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.854150 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.886678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.870867 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.890734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.888860 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.898289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.898255 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qxl\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.917622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.915189 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token\") pod \"e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:50.966990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:50.966955 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:51.289644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:51.289355 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:51.374829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:51.371905 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz"] Oct 30 13:27:51.400571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:51.399911 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:51.400571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:51.400042 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:51.400571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:51.400087 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:52.299333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.299292 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" event={"ID":"d33dd336-737c-43a5-974f-ba026ceeaf0c","Type":"ContainerStarted","Data":"d1637a72b19ea68ce7c23eab0ec3b15e085d50def8255a71f1e5b748b9a9e147"} Oct 30 13:27:52.299936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.299910 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" event={"ID":"d33dd336-737c-43a5-974f-ba026ceeaf0c","Type":"ContainerStarted","Data":"2d93d235cff5f6bc230b8c8ce4753c5058cc9649d45e1a6fb00c297833e75126"} Oct 30 13:27:52.796652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.793638 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podStartSLOduration=2.793614932 podStartE2EDuration="2.793614932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:52.391809005 +0000 UTC m=+1448.258427192" watchObservedRunningTime="2026-10-30 13:27:52.793614932 +0000 UTC m=+1448.660233107" Oct 30 13:27:52.796652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.794175 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6"] Oct 30 13:27:52.800510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.800488 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.819537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.819517 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9bf8ad-openshift-default-dockercfg-shhh4\"" Oct 30 13:27:52.923417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923373 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923440 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4c8\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923474 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923538 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923584 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923703 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923765 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923849 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.923899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.923871 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:52.970622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.970108 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc"] Oct 30 13:27:52.996659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.979492 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:52.996659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.979642 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:52.996659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.987390 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:52.996659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:52.991519 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw"] Oct 30 13:27:53.001303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.001277 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6"] Oct 30 13:27:53.001303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.001318 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc"] Oct 30 13:27:53.001643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.001418 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.009865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.009835 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df"] Oct 30 13:27:53.014071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.014048 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.025713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.025685 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.025823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.025727 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.025823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.025752 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.025941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.025921 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.025991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.025951 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026055 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026137 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026213 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026278 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026309 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026398 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026437 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4c8\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026463 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026491 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026529 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcz5\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026575 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.026657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026574 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026689 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026753 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.026833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.026786 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.027391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.027101 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.027391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.027344 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.029875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.029845 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.033261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.033217 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.085776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.085709 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.088955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.088902 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4c8\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.115817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.115544 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.115817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.115631 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df"] Oct 30 13:27:53.120346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.120324 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw"] Oct 30 13:27:53.129239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129211 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcz5\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.129364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129268 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.129364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129297 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.129364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129336 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.129539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129369 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.129539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129394 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.129539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129429 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfrj\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.129539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129486 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.129804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129566 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.129804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129617 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.129804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129647 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.129804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129688 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.129804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129718 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.129804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129768 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129863 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129899 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129927 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.129957 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130005 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.130102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130095 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.130458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130133 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.130458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130163 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.130458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130189 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.130934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.130909 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.131822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.131796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.132072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.132046 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.132342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.132317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.132686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.132661 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c95d\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.132771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.132717 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.132844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.132769 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.133481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.133454 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.139545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.139523 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.141564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.141543 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.146303 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:27:53.146254 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:27:53.156673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.156646 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.157120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.157080 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcz5\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.234541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234501 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.234782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234556 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.234782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234588 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.234782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234625 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.234782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234668 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.234782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234723 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.234782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234760 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c95d\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234793 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234882 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234924 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234966 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.234992 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfrj\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.235114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235087 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.235465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235222 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.235465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235230 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235267 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235301 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.235465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235514 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235821 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.235872 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.236068 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.236513 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.236553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.236875 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.236978 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.237932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.237589 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.238370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.238021 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.242927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.241532 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.242927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.242662 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.245118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.245081 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.257724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.257690 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c95d\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.257882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.257823 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.259066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.259039 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfrj\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.259326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.259294 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.313972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.313936 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:53.317434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.317402 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:53.318675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.318651 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:27:53.326835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.326812 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:53.417719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.416852 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:53.442954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.442928 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:53.650798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.650769 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc"] Oct 30 13:27:53.651123 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:27:53.651088 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47eff72_66ab_4409_9bd0_f801817d1a26.slice/crio-dc6e0d849266a4da286ce2d3ab7628bf8eff65efbed797c40e7cd74074fb121f WatchSource:0}: Error finding container dc6e0d849266a4da286ce2d3ab7628bf8eff65efbed797c40e7cd74074fb121f: Status 404 returned error can't find the container with id dc6e0d849266a4da286ce2d3ab7628bf8eff65efbed797c40e7cd74074fb121f Oct 30 13:27:53.654645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.654584 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.654740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.654695 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.654778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.654740 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.672107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.672085 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw"] Oct 30 13:27:53.676836 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:27:53.676799 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5f7f81_8d55_42c6_8ff6_718e868da9d6.slice/crio-0813c53dfdb6b96d0c1ea05257c06eaab849e2e53dc9e658d7da03f19783b280 WatchSource:0}: Error finding container 0813c53dfdb6b96d0c1ea05257c06eaab849e2e53dc9e658d7da03f19783b280: Status 404 returned error can't find the container with id 0813c53dfdb6b96d0c1ea05257c06eaab849e2e53dc9e658d7da03f19783b280 Oct 30 13:27:53.679436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.679399 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.679511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.679473 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.679571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.679516 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.756950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.756917 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6"] Oct 30 13:27:53.763304 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:27:53.763257 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3a03f2_e04a_4c64_af75_902820c8ff2b.slice/crio-00d4c70e00192e4f498204e852ebd02c01a94291ac1595c4ad1bd7f9bdad6712 WatchSource:0}: Error finding container 00d4c70e00192e4f498204e852ebd02c01a94291ac1595c4ad1bd7f9bdad6712: Status 404 returned error can't find the container with id 00d4c70e00192e4f498204e852ebd02c01a94291ac1595c4ad1bd7f9bdad6712 Oct 30 13:27:53.766195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.766156 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.766277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.766249 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.766329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.766298 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.823414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.822052 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df"] Oct 30 13:27:53.831295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.830280 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.831295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.830345 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:53.831295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:53.830376 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:27:54.039621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.035752 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:54.228615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.228566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:54.328003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.327965 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" event={"ID":"4f3a03f2-e04a-4c64-af75-902820c8ff2b","Type":"ContainerStarted","Data":"622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607"} Oct 30 13:27:54.328493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.328022 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" event={"ID":"4f3a03f2-e04a-4c64-af75-902820c8ff2b","Type":"ContainerStarted","Data":"00d4c70e00192e4f498204e852ebd02c01a94291ac1595c4ad1bd7f9bdad6712"} Oct 30 13:27:54.330069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.330036 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" event={"ID":"e47eff72-66ab-4409-9bd0-f801817d1a26","Type":"ContainerStarted","Data":"874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e"} Oct 30 13:27:54.330213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.330093 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" event={"ID":"e47eff72-66ab-4409-9bd0-f801817d1a26","Type":"ContainerStarted","Data":"dc6e0d849266a4da286ce2d3ab7628bf8eff65efbed797c40e7cd74074fb121f"} Oct 30 13:27:54.332042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.332011 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" event={"ID":"af5f7f81-8d55-42c6-8ff6-718e868da9d6","Type":"ContainerStarted","Data":"1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02"} Oct 30 13:27:54.332181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.332067 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" event={"ID":"af5f7f81-8d55-42c6-8ff6-718e868da9d6","Type":"ContainerStarted","Data":"0813c53dfdb6b96d0c1ea05257c06eaab849e2e53dc9e658d7da03f19783b280"} Oct 30 13:27:54.334068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.334041 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" event={"ID":"f533284d-9658-4d62-9502-43b0d7e9ce9b","Type":"ContainerStarted","Data":"49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674"} Oct 30 13:27:54.334169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.334090 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" event={"ID":"f533284d-9658-4d62-9502-43b0d7e9ce9b","Type":"ContainerStarted","Data":"1d14bf3bdef19a41d0bde8c4ee71d05b7aaae5307efa3a7642f14394cb6ffb3f"} Oct 30 13:27:54.385017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.384963 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podStartSLOduration=2.38494708 podStartE2EDuration="2.38494708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:54.381302191 +0000 UTC m=+1450.247920374" watchObservedRunningTime="2026-10-30 13:27:54.38494708 +0000 UTC m=+1450.251565253" Oct 30 13:27:54.417501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.417466 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:27:54.419410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.419372 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:27:54.419548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.419440 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:27:54.435526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.435467 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podStartSLOduration=2.435447035 podStartE2EDuration="2.435447035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:54.430274897 +0000 UTC m=+1450.296893096" watchObservedRunningTime="2026-10-30 13:27:54.435447035 +0000 UTC m=+1450.302065209" Oct 30 13:27:54.443526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.443495 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:27:54.445718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.445686 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:27:54.445841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.445738 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:27:54.481447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.481357 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podStartSLOduration=2.481342477 podStartE2EDuration="2.481342477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:54.477221153 +0000 UTC m=+1450.343839328" watchObservedRunningTime="2026-10-30 13:27:54.481342477 +0000 UTC m=+1450.347960650" Oct 30 13:27:54.524764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.524716 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podStartSLOduration=2.524703856 podStartE2EDuration="2.524703856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:54.523573829 +0000 UTC m=+1450.390192002" watchObservedRunningTime="2026-10-30 13:27:54.524703856 +0000 UTC m=+1450.391322029" Oct 30 13:27:54.780343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.779938 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:27:54.780343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:54.779941 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:27:55.011677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.010566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:55.256662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.254685 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:55.349129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.327980 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:27:55.349129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.332045 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:27:55.349129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.341756 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:27:55.349129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.341816 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:27:55.349129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.342280 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:27:55.349129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.342322 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:27:55.454625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.448117 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:27:55.454625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.448177 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:27:55.477297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.476690 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:27:55.477297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.476766 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:27:55.985625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:55.984740 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:56.050622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.049817 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:27:56.072628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.057673 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:27:56.072628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.070437 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:56.091619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.084984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:56.273735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.273631 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:56.275163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.275135 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:56.342624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.335134 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:27:56.342624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.335196 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:27:56.342624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.335976 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:27:56.342624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.336015 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:27:56.466698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.371025 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn"] Oct 30 13:27:56.466698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.371301 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" containerID="cri-o://2e84c545b2550cebba9716a6c9f8217be03ba36bfeb9f8226e5e3cb9a4916233" gracePeriod=30 Oct 30 13:27:56.584827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.507131 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:27:56.584827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.507197 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:27:56.584827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.540368 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:27:56.584827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.540427 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:27:56.710149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.686400 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 13:27:56.881347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.800909 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:27:56.952475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.897401 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:56.986669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:56.986533 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:57.008730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.004999 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:27:57.076762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.075191 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:57.080646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.078654 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:27:57.080646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.079058 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:57.114653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.106571 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:27:57.137528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.115049 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:27:57.137528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.115127 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:27:57.163647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.141614 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:57.435677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.432824 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:27:57.435677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.432903 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:27:57.528618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.516941 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:27:57.528618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.517002 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:27:57.560620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.535735 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:57.560620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.542820 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:27:57.593623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.590689 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:27:57.593623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.590752 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:27:57.608618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.606009 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:27:57.608618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.606088 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:27:57.836647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.820496 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:57.919647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.905211 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:57.919647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.905690 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:27:57.919647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.915131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:57.933214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.933170 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:27:57.941951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.941902 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:27:57.991428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:57.991300 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:58.082263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.082221 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:58.089679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.087496 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:58.112737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.112694 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:58.194625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.154616 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd"] Oct 30 13:27:58.302702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.293205 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:58.353179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.353090 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:27:58.353436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.353406 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:27:58.355343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.355317 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:27:58.362629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.355496 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:27:58.375955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.375921 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.412077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.412031 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-642765\"" Oct 30 13:27:58.446413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.446331 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:27:58.446773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.446669 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:27:58.510622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.503969 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:27:58.510622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.504036 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:27:58.547792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.536136 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-642765-d576798f4-dltfd\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.547792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.536334 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7447z\" (UniqueName: \"kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z\") pod \"payload-processing-e2e-ipp-b-642765-d576798f4-dltfd\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.582635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.581036 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:58.679262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.679172 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd"] Oct 30 13:27:58.708778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.702027 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx"] Oct 30 13:27:58.770714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.752668 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-642765-d576798f4-dltfd\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.770714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.752865 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7447z\" (UniqueName: \"kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z\") pod \"payload-processing-e2e-ipp-b-642765-d576798f4-dltfd\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.770714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.754404 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-642765-d576798f4-dltfd\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.770714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.761912 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx"] Oct 30 13:27:58.770714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.762065 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:58.857623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.853325 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7447z\" (UniqueName: \"kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z\") pod \"payload-processing-e2e-ipp-b-642765-d576798f4-dltfd\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.857623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.853737 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-642765-dockercfg-clck7\"" Oct 30 13:27:58.857623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.856038 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvnx\" (UniqueName: \"kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx\") pod \"payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:58.857623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.856254 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:58.888932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.888903 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:27:58.916694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.915558 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:58.963620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.958542 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvnx\" (UniqueName: \"kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx\") pod \"payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:58.963620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.958837 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:58.963620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:58.959842 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:59.026758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.020377 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvnx\" (UniqueName: \"kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx\") pod \"payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:59.094635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.082976 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:27:59.094635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.084076 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:27:59.094635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.091780 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 13:27:59.115487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.114719 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:27:59.276936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.276899 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:27:59.310640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.293197 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:27:59.345142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.345105 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:27:59.345401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.345370 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:27:59.346464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.346397 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:27:59.356869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.356830 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:27:59.435723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.418768 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:27:59.436025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.435986 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:27:59.616624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.605854 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:27:59.668713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.625211 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:27:59.978629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.810694 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:00.000624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:27:59.997298 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:28:00.180623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.086192 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:00.218365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.086541 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:00.248786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.248692 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:28:00.303925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.303865 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 13:28:00.336685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.319823 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:28:00.374830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.374795 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:28:00.375115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.375067 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:28:00.379805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.379774 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:28:00.400171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.400131 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:28:00.442755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.439798 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:28:00.444730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.442998 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:28:00.497679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.466224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:00.497679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.485887 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:28:00.497679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.485943 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:28:00.652323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.652197 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:00.762632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:00.761819 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd"] Oct 30 13:28:01.293670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.269098 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 13:28:01.389442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.326238 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 13:28:01.425840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.328113 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:28:01.426158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.426122 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:28:01.542619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.515367 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 13:28:01.542619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.515788 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 13:28:01.542619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.516089 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 13:28:01.542619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.518357 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:28:01.546672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.543185 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:28:01.546672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.543270 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:28:01.546672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.543391 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:28:01.546672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.543431 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:28:01.571198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.549150 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:28:01.571198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.549214 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:28:01.669906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.659456 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:01.669906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.659965 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:28:01.836581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:01.835635 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" event={"ID":"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3","Type":"ContainerStarted","Data":"37cd1015e509f37025495cb8e44f861e7bdabea5865c0f90b6594b72d75ee184"} Oct 30 13:28:02.096007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.095958 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 13:28:02.117513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.116663 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:28:02.223972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.223943 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 13:28:02.420672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.403878 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:28:02.420971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.420932 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:28:02.435563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.427880 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:28:02.435563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.427935 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:28:02.454759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.451294 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:02.487391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.487100 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:28:02.505628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.501581 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:28:02.505628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.488813 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:28:02.505628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.501694 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:28:02.595191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.594568 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:28:02.595738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.595718 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 13:28:02.696361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.696277 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:28:02.817056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.817021 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx"] Oct 30 13:28:02.896543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.896510 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" event={"ID":"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3","Type":"ContainerStarted","Data":"7d069da6e678a97d4bc181f5b3693c8c34d41b29608f99c9376a3e822f23faa8"} Oct 30 13:28:02.896957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.896936 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:28:02.914647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.914572 5693 generic.go:356] "Generic (PLEG): container finished" podID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerID="2e84c545b2550cebba9716a6c9f8217be03ba36bfeb9f8226e5e3cb9a4916233" exitCode=0 Oct 30 13:28:02.914959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.914933 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" event={"ID":"cc479868-520a-45f0-bc7c-ba06a8f95830","Type":"ContainerDied","Data":"2e84c545b2550cebba9716a6c9f8217be03ba36bfeb9f8226e5e3cb9a4916233"} Oct 30 13:28:02.915061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.915000 5693 scope.go:122] "RemoveContainer" containerID="ea60ef3bb25e9934a7526f1bec552506158275661eb2419b9d7a5f5c3b0698d5" Oct 30 13:28:02.946167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:02.946109 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" podStartSLOduration=5.9460879030000005 podStartE2EDuration="5.946087903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:02.942970496 +0000 UTC m=+1458.809588670" watchObservedRunningTime="2026-10-30 13:28:02.946087903 +0000 UTC m=+1458.812706078" Oct 30 13:28:03.024631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.024576 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:28:03.208672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.208946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208678 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.208946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208743 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.208946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208817 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.208946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208854 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.208946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208895 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.208946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.208932 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.209865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.209022 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.209865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.209062 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token\") pod \"cc479868-520a-45f0-bc7c-ba06a8f95830\" (UID: \"cc479868-520a-45f0-bc7c-ba06a8f95830\") " Oct 30 13:28:03.213186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.213157 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:03.213525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.213504 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:03.213922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.213902 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:03.214381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.214360 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:03.223650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.217389 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:03.223650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.217650 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:03.224412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.224156 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:03.226800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.226215 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "kube-api-access-hhwdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:03.256084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.255684 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo" pod "cc479868-520a-45f0-bc7c-ba06a8f95830" (UID: "cc479868-520a-45f0-bc7c-ba06a8f95830"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:28:03.327317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.326124 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.327317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.326171 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc479868-520a-45f0-bc7c-ba06a8f95830-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.327317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.326195 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.327317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.326218 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.327317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.326241 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.340304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.339279 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:28:03.340304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.339378 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:28:03.341223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.341126 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.341223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.341153 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc479868-520a-45f0-bc7c-ba06a8f95830-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.341223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.341177 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwdp\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-kube-api-access-hhwdp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.341223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.341192 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc479868-520a-45f0-bc7c-ba06a8f95830-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:03.342971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.342811 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:03.425585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.425549 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:28:03.425900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.425870 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:28:03.443805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.443764 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:28:03.443975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.443844 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:28:03.932266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.932227 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" event={"ID":"cc479868-520a-45f0-bc7c-ba06a8f95830","Type":"ContainerDied","Data":"17ff5af89e65a0b8b7c7cfc6e5c2d5ff7fbe19d869670b3384a01203d98d2451"} Oct 30 13:28:03.932934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.932911 5693 scope.go:122] "RemoveContainer" containerID="2e84c545b2550cebba9716a6c9f8217be03ba36bfeb9f8226e5e3cb9a4916233" Oct 30 13:28:03.933305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.933279 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn" Oct 30 13:28:03.942527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.942488 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" event={"ID":"31fa3123-b4f7-4edc-baa5-14d1f430741d","Type":"ContainerStarted","Data":"d879e60fcb527b5c988bcfc349daa1b441f1a7b969f29515b578c5b0b069e45f"} Oct 30 13:28:03.942895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:03.942872 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" event={"ID":"31fa3123-b4f7-4edc-baa5-14d1f430741d","Type":"ContainerStarted","Data":"eb430af7d59c8da0e179cb9f273732427578e6580fcd00dbfdf2f6e29c6a9b71"} Oct 30 13:28:04.018964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.018888 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" podStartSLOduration=7.018867931 podStartE2EDuration="7.018867931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:04.005064382 +0000 UTC m=+1459.871682558" watchObservedRunningTime="2026-10-30 13:28:04.018867931 +0000 UTC m=+1459.885486105" Oct 30 13:28:04.065177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.065145 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:28:04.065456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.065436 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn"] Oct 30 13:28:04.067085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.067064 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:28:04.068305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.068284 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:28:04.070525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.070491 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d797fc-openshift-default-7f5f87b58f-kjfcn"] Oct 30 13:28:04.315346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.311007 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" path="/var/lib/kubelet/pods/cc479868-520a-45f0-bc7c-ba06a8f95830/volumes" Oct 30 13:28:04.338533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.338485 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:28:04.339166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.339141 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:28:04.340344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.340321 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:28:04.658784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.658697 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:28:04.659723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.659694 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:28:04.660672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.660653 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:28:04.756733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.756683 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:28:04.757894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.757868 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:28:04.759150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.759118 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:28:04.760810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.760786 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:28:04.761220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.761195 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:28:04.950280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:04.950183 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:28:05.229733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:05.229646 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:06.910819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:06.910771 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:07.647394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:07.647350 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7"] Oct 30 13:28:07.676958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:07.676927 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn"] Oct 30 13:28:07.809709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:07.809671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:07.953828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:07.953036 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:08.730846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:08.730502 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:08.967370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:08.967342 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:28:09.002086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.002059 5693 generic.go:356] "Generic (PLEG): container finished" podID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerID="9f0d325304a2cc2c716b0a666f9613493f3ad14e90f6db9fe5ce745e122294fd" exitCode=0 Oct 30 13:28:09.002219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.002139 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" event={"ID":"fd0653aa-d044-4ba8-9467-2ada5bd41b14","Type":"ContainerDied","Data":"9f0d325304a2cc2c716b0a666f9613493f3ad14e90f6db9fe5ce745e122294fd"} Oct 30 13:28:09.002219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.002197 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" event={"ID":"fd0653aa-d044-4ba8-9467-2ada5bd41b14","Type":"ContainerDied","Data":"1408eec01b9147f45271ab9a7c5d81793d9ee216b04d2ef02db1652a16b25472"} Oct 30 13:28:09.002327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.002222 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1408eec01b9147f45271ab9a7c5d81793d9ee216b04d2ef02db1652a16b25472" Oct 30 13:28:09.003656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.003632 5693 generic.go:356] "Generic (PLEG): container finished" podID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerID="ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd" exitCode=0 Oct 30 13:28:09.003768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.003674 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" Oct 30 13:28:09.003768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.003684 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" event={"ID":"62743f5d-7cf4-431f-9b73-ee73f11b9d42","Type":"ContainerDied","Data":"ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd"} Oct 30 13:28:09.003768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.003726 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn" event={"ID":"62743f5d-7cf4-431f-9b73-ee73f11b9d42","Type":"ContainerDied","Data":"7e71b3c51d28b0cdc828a3c0fce475b992bcc98ef79ffdec799494c4428f271a"} Oct 30 13:28:09.003768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.003747 5693 scope.go:122] "RemoveContainer" containerID="ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd" Oct 30 13:28:09.013009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.012990 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:28:09.015226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.015206 5693 scope.go:122] "RemoveContainer" containerID="ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd" Oct 30 13:28:09.015508 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:09.015482 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd\": container with ID starting with ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd not found: ID does not exist" containerID="ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd" Oct 30 13:28:09.015610 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.015518 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd"} err="failed to get container status \"ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd\": rpc error: code = NotFound desc = could not find container \"ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd\": container with ID starting with ea4b418afb260c95a443bdbe961c798d80130d9658e97aeda4e4f38993c799cd not found: ID does not exist" Oct 30 13:28:09.103860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.103826 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume\") pod \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " Oct 30 13:28:09.104060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.103984 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj\" (UniqueName: \"kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj\") pod \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " Oct 30 13:28:09.104130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.104066 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6\" (UniqueName: \"kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6\") pod \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\" (UID: \"fd0653aa-d044-4ba8-9467-2ada5bd41b14\") " Oct 30 13:28:09.104187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.104160 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume\") pod \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\" (UID: \"62743f5d-7cf4-431f-9b73-ee73f11b9d42\") " Oct 30 13:28:09.104304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.104270 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume" pod "fd0653aa-d044-4ba8-9467-2ada5bd41b14" (UID: "fd0653aa-d044-4ba8-9467-2ada5bd41b14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:09.104806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.104779 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume" pod "62743f5d-7cf4-431f-9b73-ee73f11b9d42" (UID: "62743f5d-7cf4-431f-9b73-ee73f11b9d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:09.105001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.104966 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62743f5d-7cf4-431f-9b73-ee73f11b9d42-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:09.105001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.104983 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0653aa-d044-4ba8-9467-2ada5bd41b14-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:09.106936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.106909 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj" pod "62743f5d-7cf4-431f-9b73-ee73f11b9d42" (UID: "62743f5d-7cf4-431f-9b73-ee73f11b9d42"). InnerVolumeSpecName "kube-api-access-nwcxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:09.107499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.107466 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6" pod "fd0653aa-d044-4ba8-9467-2ada5bd41b14" (UID: "fd0653aa-d044-4ba8-9467-2ada5bd41b14"). InnerVolumeSpecName "kube-api-access-wb7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:09.206286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.206208 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7b6\" (UniqueName: \"kubernetes.io/projected/fd0653aa-d044-4ba8-9467-2ada5bd41b14-kube-api-access-wb7b6\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:09.206286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.206236 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcxj\" (UniqueName: \"kubernetes.io/projected/62743f5d-7cf4-431f-9b73-ee73f11b9d42-kube-api-access-nwcxj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:09.355059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.355027 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn"] Oct 30 13:28:09.378027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:09.376354 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5cee02-64746d775b-kgvvn"] Oct 30 13:28:10.011145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:10.011115 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj" Oct 30 13:28:10.071343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:10.071310 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj"] Oct 30 13:28:10.083090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:10.083058 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5cee02-6665466547-s2xxj"] Oct 30 13:28:10.256577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:10.256541 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" path="/var/lib/kubelet/pods/62743f5d-7cf4-431f-9b73-ee73f11b9d42/volumes" Oct 30 13:28:10.257068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:10.257052 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" path="/var/lib/kubelet/pods/fd0653aa-d044-4ba8-9467-2ada5bd41b14/volumes" Oct 30 13:28:10.430144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:10.427024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:12.134944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:12.134907 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:12.647715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:12.647648 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" podUID="afecc174-c237-4352-a666-a7b3b56b5428" containerName="payload-processing" containerID="cri-o://241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c" gracePeriod=25 Oct 30 13:28:12.677219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:12.677153 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" podUID="90c5979c-50ca-4258-bf72-01336e03314f" containerName="payload-pre-processing" containerID="cri-o://050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e" gracePeriod=25 Oct 30 13:28:12.683483 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:12.683452 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafecc174_c237_4352_a666_a7b3b56b5428.slice/crio-conmon-241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafecc174_c237_4352_a666_a7b3b56b5428.slice/crio-241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:28:12.967816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:12.967780 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:28:12.990034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:12.990010 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:28:13.033544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.033506 5693 generic.go:356] "Generic (PLEG): container finished" podID="afecc174-c237-4352-a666-a7b3b56b5428" containerID="241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c" exitCode=0 Oct 30 13:28:13.033755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.033718 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" Oct 30 13:28:13.033829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.033765 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" event={"ID":"afecc174-c237-4352-a666-a7b3b56b5428","Type":"ContainerDied","Data":"241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c"} Oct 30 13:28:13.033896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.033857 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7" event={"ID":"afecc174-c237-4352-a666-a7b3b56b5428","Type":"ContainerDied","Data":"ba73c1e46a095fe6c09d224b3353722d46605a312266ba399b3bd01c2788ffda"} Oct 30 13:28:13.033896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.033890 5693 scope.go:122] "RemoveContainer" containerID="241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c" Oct 30 13:28:13.035738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.035361 5693 generic.go:356] "Generic (PLEG): container finished" podID="90c5979c-50ca-4258-bf72-01336e03314f" containerID="050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e" exitCode=0 Oct 30 13:28:13.035738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.035441 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" event={"ID":"90c5979c-50ca-4258-bf72-01336e03314f","Type":"ContainerDied","Data":"050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e"} Oct 30 13:28:13.035738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.035517 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" event={"ID":"90c5979c-50ca-4258-bf72-01336e03314f","Type":"ContainerDied","Data":"6e2c635a9f9c0b9e928288de5a64b2f64a41254b086bc371137e4f903adf5785"} Oct 30 13:28:13.035738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.035614 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn" Oct 30 13:28:13.051488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.051454 5693 scope.go:122] "RemoveContainer" containerID="241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c" Oct 30 13:28:13.052409 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:13.052379 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c\": container with ID starting with 241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c not found: ID does not exist" containerID="241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c" Oct 30 13:28:13.052518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.052421 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c"} err="failed to get container status \"241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c\": rpc error: code = NotFound desc = could not find container \"241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c\": container with ID starting with 241fe7cf316e6f38a28dfabe7e952c8ff1e687a5412113215733479fb0cb438c not found: ID does not exist" Oct 30 13:28:13.052518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.052443 5693 scope.go:122] "RemoveContainer" containerID="050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e" Oct 30 13:28:13.063942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.063924 5693 scope.go:122] "RemoveContainer" containerID="050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e" Oct 30 13:28:13.064267 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:13.064244 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e\": container with ID starting with 050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e not found: ID does not exist" containerID="050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e" Oct 30 13:28:13.064343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.064275 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e"} err="failed to get container status \"050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e\": rpc error: code = NotFound desc = could not find container \"050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e\": container with ID starting with 050bb5e2b9b1192cd0f2c3d964b61984b918b455818c1a7241c22786f55c167e not found: ID does not exist" Oct 30 13:28:13.077627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.077573 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc\" (UniqueName: \"kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc\") pod \"90c5979c-50ca-4258-bf72-01336e03314f\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " Oct 30 13:28:13.077747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.077736 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume\") pod \"afecc174-c237-4352-a666-a7b3b56b5428\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " Oct 30 13:28:13.077897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.077880 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume\") pod \"90c5979c-50ca-4258-bf72-01336e03314f\" (UID: \"90c5979c-50ca-4258-bf72-01336e03314f\") " Oct 30 13:28:13.078022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.077951 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9\" (UniqueName: \"kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9\") pod \"afecc174-c237-4352-a666-a7b3b56b5428\" (UID: \"afecc174-c237-4352-a666-a7b3b56b5428\") " Oct 30 13:28:13.078299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.078266 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume" pod "afecc174-c237-4352-a666-a7b3b56b5428" (UID: "afecc174-c237-4352-a666-a7b3b56b5428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:13.078299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.078270 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume" pod "90c5979c-50ca-4258-bf72-01336e03314f" (UID: "90c5979c-50ca-4258-bf72-01336e03314f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:13.078572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.078545 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afecc174-c237-4352-a666-a7b3b56b5428-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:13.078700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.078576 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5979c-50ca-4258-bf72-01336e03314f-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:13.079877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.079855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc" pod "90c5979c-50ca-4258-bf72-01336e03314f" (UID: "90c5979c-50ca-4258-bf72-01336e03314f"). InnerVolumeSpecName "kube-api-access-qcssc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:13.080431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.080409 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9" pod "afecc174-c237-4352-a666-a7b3b56b5428" (UID: "afecc174-c237-4352-a666-a7b3b56b5428"). InnerVolumeSpecName "kube-api-access-77zt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:13.180435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.180398 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zt9\" (UniqueName: \"kubernetes.io/projected/afecc174-c237-4352-a666-a7b3b56b5428-kube-api-access-77zt9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:13.180435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.180440 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcssc\" (UniqueName: \"kubernetes.io/projected/90c5979c-50ca-4258-bf72-01336e03314f-kube-api-access-qcssc\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:13.443622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.436666 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7"] Oct 30 13:28:13.475766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.475689 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b35c0f-6b7999b996-22xx7"] Oct 30 13:28:13.507565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.505646 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn"] Oct 30 13:28:13.510399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.510360 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b35c0f-fd6d848ff-zbzdn"] Oct 30 13:28:13.972675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:13.967453 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:28:14.333625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:14.331931 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c5979c-50ca-4258-bf72-01336e03314f" path="/var/lib/kubelet/pods/90c5979c-50ca-4258-bf72-01336e03314f/volumes" Oct 30 13:28:14.333625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:14.332530 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afecc174-c237-4352-a666-a7b3b56b5428" path="/var/lib/kubelet/pods/afecc174-c237-4352-a666-a7b3b56b5428/volumes" Oct 30 13:28:15.193622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.193507 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:15.371622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.358666 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx"] Oct 30 13:28:15.378492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.378459 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" Oct 30 13:28:15.378715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.378692 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" Oct 30 13:28:15.378856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.378843 5693 cpu_manager.go:414] "containerMap: removing container" podUID="90c5979c-50ca-4258-bf72-01336e03314f" containerName="payload-pre-processing" Oct 30 13:28:15.378961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.378947 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c5979c-50ca-4258-bf72-01336e03314f" containerName="payload-pre-processing" Oct 30 13:28:15.379079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.379067 5693 cpu_manager.go:414] "containerMap: removing container" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerName="payload-processing" Oct 30 13:28:15.379176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.379165 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerName="payload-processing" Oct 30 13:28:15.379370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.379353 5693 cpu_manager.go:414] "containerMap: removing container" podUID="afecc174-c237-4352-a666-a7b3b56b5428" containerName="payload-processing" Oct 30 13:28:15.379490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.379475 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afecc174-c237-4352-a666-a7b3b56b5428" containerName="payload-processing" Oct 30 13:28:15.379632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.379619 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerName="payload-pre-processing" Oct 30 13:28:15.379744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.379730 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerName="payload-pre-processing" Oct 30 13:28:15.380217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.380198 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0653aa-d044-4ba8-9467-2ada5bd41b14" containerName="payload-pre-processing" Oct 30 13:28:15.380341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.380329 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" Oct 30 13:28:15.380442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.380431 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c5979c-50ca-4258-bf72-01336e03314f" containerName="payload-pre-processing" Oct 30 13:28:15.380548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.380537 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="62743f5d-7cf4-431f-9b73-ee73f11b9d42" containerName="payload-processing" Oct 30 13:28:15.380664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.380652 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="afecc174-c237-4352-a666-a7b3b56b5428" containerName="payload-processing" Oct 30 13:28:15.408410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.408378 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx"] Oct 30 13:28:15.408857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.408769 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.490720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.482892 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-642765-serving-cert\"" Oct 30 13:28:15.633647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.632381 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.633647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.632451 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.633647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.632588 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.633647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.632693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppztg\" (UniqueName: \"kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.661633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.660591 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:15.745669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.737949 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.745669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.738045 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.745669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.738200 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.745669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.738292 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppztg\" (UniqueName: \"kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.836673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.829466 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:15.859105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:15.859053 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:16.072676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.067571 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:16.097724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.083343 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:16.122694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.094212 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.123271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.121919 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.311651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.126623 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.312494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.130706 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.313192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.140948 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:28:16.335124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.234894 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.335502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.311129 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppztg\" (UniqueName: \"kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg\") pod \"maas-api-e2e-ipp-b-642765-5d8644f549-59tcx\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:16.492960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.492915 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.524855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.524825 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:16.622797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.622696 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:16.683126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.683034 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:28:16.816952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:16.813663 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 13:28:17.391284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:17.391249 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx"] Oct 30 13:28:17.394362 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:28:17.394329 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65836f0_a900_49e6_a446_d24d28d0c55e.slice/crio-5a9188d412541ad939cdf874b0a2fe68ec58948b7ae96993e211dd335a458dda WatchSource:0}: Error finding container 5a9188d412541ad939cdf874b0a2fe68ec58948b7ae96993e211dd335a458dda: Status 404 returned error can't find the container with id 5a9188d412541ad939cdf874b0a2fe68ec58948b7ae96993e211dd335a458dda Oct 30 13:28:17.553300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:17.553262 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" event={"ID":"d65836f0-a900-49e6-a446-d24d28d0c55e","Type":"ContainerStarted","Data":"5a9188d412541ad939cdf874b0a2fe68ec58948b7ae96993e211dd335a458dda"} Oct 30 13:28:17.854621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:17.843291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:18.563514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:18.563473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" event={"ID":"d65836f0-a900-49e6-a446-d24d28d0c55e","Type":"ContainerStarted","Data":"1dc17deea1089ef6af877d7cb6fd775095e63a694956a4c70d99250c55aaa379"} Oct 30 13:28:18.563932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:18.563573 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:18.604361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:18.604311 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" podStartSLOduration=3.202343012 podStartE2EDuration="3.604297828s" totalImagesPullingTime="401.954816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:18.601477177 +0000 UTC m=+1474.468095350" watchObservedRunningTime="2026-10-30 13:28:18.604297828 +0000 UTC m=+1474.470916016" Oct 30 13:28:19.528317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:19.527253 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:20.788187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:20.788140 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg"] Oct 30 13:28:20.788692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:20.788435 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" podUID="da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" containerName="maas-api" containerID="cri-o://9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07" gracePeriod=30 Oct 30 13:28:20.893493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:20.893452 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:21.278619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.275638 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:28:21.315626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.315048 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls\") pod \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " Oct 30 13:28:21.315626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.315131 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs\") pod \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " Oct 30 13:28:21.315626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.315184 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7\" (UniqueName: \"kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7\") pod \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " Oct 30 13:28:21.315626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.315216 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca\") pod \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\" (UID: \"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796\") " Oct 30 13:28:21.318421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.318033 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca" pod "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" (UID: "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:21.361032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.360980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls" pod "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" (UID: "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:21.371390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.370807 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7" pod "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" (UID: "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796"). InnerVolumeSpecName "kube-api-access-vcfh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:21.371390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.370903 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs" pod "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" (UID: "da3e6d59-ddc6-4b7a-8df4-ec475f8dd796"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:21.418718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.418635 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.418718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.418669 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.418718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.418683 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfh7\" (UniqueName: \"kubernetes.io/projected/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-kube-api-access-vcfh7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.418718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.418712 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.485444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.485411 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:21.594184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.594152 5693 generic.go:356] "Generic (PLEG): container finished" podID="da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" containerID="9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07" exitCode=0 Oct 30 13:28:21.594371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.594196 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" Oct 30 13:28:21.594371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.594230 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" event={"ID":"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796","Type":"ContainerDied","Data":"9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07"} Oct 30 13:28:21.594371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.594287 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg" event={"ID":"da3e6d59-ddc6-4b7a-8df4-ec475f8dd796","Type":"ContainerDied","Data":"c0af160cfa6aa65042706fd5b021fb0ccfbc2a0c0cc15a206fefee41a452d8c1"} Oct 30 13:28:21.594371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.594313 5693 scope.go:122] "RemoveContainer" containerID="9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07" Oct 30 13:28:21.605735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.605707 5693 scope.go:122] "RemoveContainer" containerID="9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07" Oct 30 13:28:21.606128 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:21.606101 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07\": container with ID starting with 9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07 not found: ID does not exist" containerID="9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07" Oct 30 13:28:21.606242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.606137 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07"} err="failed to get container status \"9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07\": rpc error: code = NotFound desc = could not find container \"9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07\": container with ID starting with 9d9e067371c12cea354cb1bc3bd21b9b994b71c7178dd87f1269a03debc3de07 not found: ID does not exist" Oct 30 13:28:21.621776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.621754 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:21.644982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.644953 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg"] Oct 30 13:28:21.648213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.648193 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b35c0f-69df4f45bb-2qwqg"] Oct 30 13:28:21.800621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:21.799449 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:22.135831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:22.135746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:22.269319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:22.269288 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" path="/var/lib/kubelet/pods/da3e6d59-ddc6-4b7a-8df4-ec475f8dd796/volumes" Oct 30 13:28:23.055250 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:23.055214 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:28:23.520966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:23.520921 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:24.585974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:24.585947 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:28:25.220241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:25.218074 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:26.622034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:26.621997 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.147579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:27.147542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.484517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:27.484485 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.713868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:27.713832 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.883792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:27.883707 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:29.385176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:29.384804 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:30.308796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:30.308760 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq"] Oct 30 13:28:30.308997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:30.308976 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" containerID="cri-o://185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743" gracePeriod=30 Oct 30 13:28:30.713947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:30.713819 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:30.865833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:30.865796 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:31.001464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:31.001426 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t"] Oct 30 13:28:31.001755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:31.001698 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" containerID="cri-o://b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25" gracePeriod=30 Oct 30 13:28:31.094543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:31.094485 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:31.239627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:31.239135 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:32.773752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:32.773722 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh"] Oct 30 13:28:32.892689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:32.892648 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:33.416837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:33.416781 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:35.109076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.109042 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:35.591693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.591669 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:28:35.675894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.675858 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676062 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676111 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676154 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676195 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676225 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676312 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676352 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.676498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.676483 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket\") pod \"860ef2b5-d62c-4059-90e9-e59a2019141c\" (UID: \"860ef2b5-d62c-4059-90e9-e59a2019141c\") " Oct 30 13:28:35.677380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.677351 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:35.678377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.678351 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:35.678640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.678547 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:35.678792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.678583 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:35.678906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.678822 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:35.680905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.680865 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:28:35.681375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.681347 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:35.681519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.681370 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "kube-api-access-9d6sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:35.682139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.682104 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy" pod "860ef2b5-d62c-4059-90e9-e59a2019141c" (UID: "860ef2b5-d62c-4059-90e9-e59a2019141c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:35.699447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.699421 5693 generic.go:356] "Generic (PLEG): container finished" podID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerID="185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743" exitCode=0 Oct 30 13:28:35.699558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.699485 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" Oct 30 13:28:35.699558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.699500 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" event={"ID":"860ef2b5-d62c-4059-90e9-e59a2019141c","Type":"ContainerDied","Data":"185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743"} Oct 30 13:28:35.699558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.699557 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq" event={"ID":"860ef2b5-d62c-4059-90e9-e59a2019141c","Type":"ContainerDied","Data":"25e247f9968fd11083869dd0eabc85846b5357326dcf7f04819537c375f22f97"} Oct 30 13:28:35.699724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.699579 5693 scope.go:122] "RemoveContainer" containerID="185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743" Oct 30 13:28:35.716568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.716550 5693 scope.go:122] "RemoveContainer" containerID="185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743" Oct 30 13:28:35.716868 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:35.716849 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743\": container with ID starting with 185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743 not found: ID does not exist" containerID="185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743" Oct 30 13:28:35.716920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.716877 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743"} err="failed to get container status \"185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743\": rpc error: code = NotFound desc = could not find container \"185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743\": container with ID starting with 185acadab462fb137e13be0c5a157f660cc43eff569c97da7c4994f137b8f743 not found: ID does not exist" Oct 30 13:28:35.751126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.751099 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq"] Oct 30 13:28:35.754990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.754963 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-d2ljq"] Oct 30 13:28:35.779202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779163 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779197 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779210 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779222 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779238 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779253 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860ef2b5-d62c-4059-90e9-e59a2019141c-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779266 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6sp\" (UniqueName: \"kubernetes.io/projected/860ef2b5-d62c-4059-90e9-e59a2019141c-kube-api-access-9d6sp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779279 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860ef2b5-d62c-4059-90e9-e59a2019141c-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:35.779425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:35.779300 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860ef2b5-d62c-4059-90e9-e59a2019141c-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.262205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.262173 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" path="/var/lib/kubelet/pods/860ef2b5-d62c-4059-90e9-e59a2019141c/volumes" Oct 30 13:28:36.275094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.275009 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:28:36.286264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286239 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286330 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286416 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286493 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286607 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286540 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286607 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286640 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286676 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286719 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs\") pod \"4e947291-0a68-48df-b65a-d3eeda038447\" (UID: \"4e947291-0a68-48df-b65a-d3eeda038447\") " Oct 30 13:28:36.286876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286852 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:36.286942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.286871 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:36.287184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287136 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:36.287279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287226 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:36.287395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287358 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:36.287635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287615 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.287784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287638 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.287784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287655 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.287784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287670 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e947291-0a68-48df-b65a-d3eeda038447-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.287784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.287681 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.288971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.288949 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:28:36.289059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.288986 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "kube-api-access-qfkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:36.289164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.289144 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:36.289265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.289250 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy" pod "4e947291-0a68-48df-b65a-d3eeda038447" (UID: "4e947291-0a68-48df-b65a-d3eeda038447"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:36.388959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.388449 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e947291-0a68-48df-b65a-d3eeda038447-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.388959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.388926 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.388959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.388943 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e947291-0a68-48df-b65a-d3eeda038447-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.389263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.388957 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkph\" (UniqueName: \"kubernetes.io/projected/4e947291-0a68-48df-b65a-d3eeda038447-kube-api-access-qfkph\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:36.709218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.709129 5693 generic.go:356] "Generic (PLEG): container finished" podID="4e947291-0a68-48df-b65a-d3eeda038447" containerID="b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25" exitCode=0 Oct 30 13:28:36.709218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.709196 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" Oct 30 13:28:36.709411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.709213 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" event={"ID":"4e947291-0a68-48df-b65a-d3eeda038447","Type":"ContainerDied","Data":"b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25"} Oct 30 13:28:36.709411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.709294 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t" event={"ID":"4e947291-0a68-48df-b65a-d3eeda038447","Type":"ContainerDied","Data":"eca115ab4e39f779b9b8c94025df12d189e754eff0cfebb2a9d48fec818792df"} Oct 30 13:28:36.709411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.709318 5693 scope.go:122] "RemoveContainer" containerID="b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25" Oct 30 13:28:36.720154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.720134 5693 scope.go:122] "RemoveContainer" containerID="ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9" Oct 30 13:28:36.730378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.730340 5693 scope.go:122] "RemoveContainer" containerID="b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25" Oct 30 13:28:36.730697 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:36.730671 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25\": container with ID starting with b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25 not found: ID does not exist" containerID="b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25" Oct 30 13:28:36.730794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.730707 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25"} err="failed to get container status \"b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25\": rpc error: code = NotFound desc = could not find container \"b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25\": container with ID starting with b24e93c1c4c08245605dfb7df8985d03a36c036ae5c65a0bbdf84589dfd8cb25 not found: ID does not exist" Oct 30 13:28:36.730794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.730725 5693 scope.go:122] "RemoveContainer" containerID="ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9" Oct 30 13:28:36.731012 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:36.730994 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9\": container with ID starting with ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9 not found: ID does not exist" containerID="ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9" Oct 30 13:28:36.731073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.731017 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9"} err="failed to get container status \"ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9\": rpc error: code = NotFound desc = could not find container \"ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9\": container with ID starting with ac0d7de88daea7e7b109db03919c367ab1f0faa99e8b9210b4d4e93c39e50ee9 not found: ID does not exist" Oct 30 13:28:36.791659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.785648 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t"] Oct 30 13:28:36.817623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:36.813123 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b35c0f-openshift-default-7c4d67946f-k894t"] Oct 30 13:28:37.045364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:37.045313 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:38.276661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:38.270617 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e947291-0a68-48df-b65a-d3eeda038447" path="/var/lib/kubelet/pods/4e947291-0a68-48df-b65a-d3eeda038447/volumes" Oct 30 13:28:39.122677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:39.122638 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:40.657697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:40.657655 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd"] Oct 30 13:28:40.669749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:40.669717 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx"] Oct 30 13:28:40.820629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:40.820586 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:40.969999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:40.969928 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:41.120100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:41.120067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:42.860754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:42.860677 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:44.566238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:44.566196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:45.658665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.658614 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" podUID="4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" containerName="payload-processing" containerID="cri-o://7d069da6e678a97d4bc181f5b3693c8c34d41b29608f99c9376a3e822f23faa8" gracePeriod=25 Oct 30 13:28:45.670392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.670348 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" podUID="31fa3123-b4f7-4edc-baa5-14d1f430741d" containerName="payload-pre-processing" containerID="cri-o://d879e60fcb527b5c988bcfc349daa1b441f1a7b969f29515b578c5b0b069e45f" gracePeriod=25 Oct 30 13:28:45.833344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.833302 5693 generic.go:356] "Generic (PLEG): container finished" podID="4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" containerID="7d069da6e678a97d4bc181f5b3693c8c34d41b29608f99c9376a3e822f23faa8" exitCode=0 Oct 30 13:28:45.833511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.833428 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" event={"ID":"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3","Type":"ContainerDied","Data":"7d069da6e678a97d4bc181f5b3693c8c34d41b29608f99c9376a3e822f23faa8"} Oct 30 13:28:45.835331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.835134 5693 generic.go:356] "Generic (PLEG): container finished" podID="31fa3123-b4f7-4edc-baa5-14d1f430741d" containerID="d879e60fcb527b5c988bcfc349daa1b441f1a7b969f29515b578c5b0b069e45f" exitCode=0 Oct 30 13:28:45.835331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.835288 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" event={"ID":"31fa3123-b4f7-4edc-baa5-14d1f430741d","Type":"ContainerDied","Data":"d879e60fcb527b5c988bcfc349daa1b441f1a7b969f29515b578c5b0b069e45f"} Oct 30 13:28:45.997737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:45.997709 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:28:46.019971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.019948 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:28:46.114505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.113976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume\") pod \"31fa3123-b4f7-4edc-baa5-14d1f430741d\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " Oct 30 13:28:46.114505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.114037 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx\" (UniqueName: \"kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx\") pod \"31fa3123-b4f7-4edc-baa5-14d1f430741d\" (UID: \"31fa3123-b4f7-4edc-baa5-14d1f430741d\") " Oct 30 13:28:46.114505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.114205 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z\" (UniqueName: \"kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z\") pod \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " Oct 30 13:28:46.114505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.114391 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume\") pod \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\" (UID: \"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3\") " Oct 30 13:28:46.114505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.114468 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume" pod "31fa3123-b4f7-4edc-baa5-14d1f430741d" (UID: "31fa3123-b4f7-4edc-baa5-14d1f430741d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:46.114953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.114910 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume" pod "4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" (UID: "4a7f62f9-35ea-40c7-91c6-a65fcd275cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:46.115456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.115428 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fa3123-b4f7-4edc-baa5-14d1f430741d-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:46.115582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.115459 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:46.117231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.117202 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z" pod "4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" (UID: "4a7f62f9-35ea-40c7-91c6-a65fcd275cb3"). InnerVolumeSpecName "kube-api-access-7447z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:46.117484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.117454 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx" pod "31fa3123-b4f7-4edc-baa5-14d1f430741d" (UID: "31fa3123-b4f7-4edc-baa5-14d1f430741d"). InnerVolumeSpecName "kube-api-access-zmvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:46.193421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.193315 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:46.235489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.224145 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvnx\" (UniqueName: \"kubernetes.io/projected/31fa3123-b4f7-4edc-baa5-14d1f430741d-kube-api-access-zmvnx\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:46.235489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.224182 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7447z\" (UniqueName: \"kubernetes.io/projected/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3-kube-api-access-7447z\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:46.844980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.844944 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" Oct 30 13:28:46.845484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.844977 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd" event={"ID":"4a7f62f9-35ea-40c7-91c6-a65fcd275cb3","Type":"ContainerDied","Data":"37cd1015e509f37025495cb8e44f861e7bdabea5865c0f90b6594b72d75ee184"} Oct 30 13:28:46.845484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.845052 5693 scope.go:122] "RemoveContainer" containerID="7d069da6e678a97d4bc181f5b3693c8c34d41b29608f99c9376a3e822f23faa8" Oct 30 13:28:46.847215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.846924 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" event={"ID":"31fa3123-b4f7-4edc-baa5-14d1f430741d","Type":"ContainerDied","Data":"eb430af7d59c8da0e179cb9f273732427578e6580fcd00dbfdf2f6e29c6a9b71"} Oct 30 13:28:46.847215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.846971 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" Oct 30 13:28:46.856525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.856506 5693 scope.go:122] "RemoveContainer" containerID="d879e60fcb527b5c988bcfc349daa1b441f1a7b969f29515b578c5b0b069e45f" Oct 30 13:28:46.895607 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.895563 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd"] Oct 30 13:28:46.907070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.907046 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-d576798f4-dltfd"] Oct 30 13:28:46.932220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.932164 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx"] Oct 30 13:28:46.936520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.936497 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx"] Oct 30 13:28:46.957118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:46.957088 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-965c4df4c-lwcfx" podUID="31fa3123-b4f7-4edc-baa5-14d1f430741d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.157:50052\" within 1s: context deadline exceeded" Oct 30 13:28:46.957254 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:28:46.957182 5693 logging.go:55] [core] [Channel #3492 SubChannel #3493]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.157:50052", ServerName: "10.134.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.157:50052: operation was canceled" Oct 30 13:28:47.378806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:47.378775 5693 scope.go:122] "RemoveContainer" containerID="c6eaa386651749a18440b4dd6a52eeae42b120998c002399f160442ace67e7ad" Oct 30 13:28:47.796014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:47.795973 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" containerID="cri-o://922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372" gracePeriod=45 Oct 30 13:28:48.018658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.018592 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:48.152664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.152638 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:28:48.258579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.258548 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fa3123-b4f7-4edc-baa5-14d1f430741d" path="/var/lib/kubelet/pods/31fa3123-b4f7-4edc-baa5-14d1f430741d/volumes" Oct 30 13:28:48.259166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.259148 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" path="/var/lib/kubelet/pods/4a7f62f9-35ea-40c7-91c6-a65fcd275cb3/volumes" Oct 30 13:28:48.264347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264319 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x\" (UniqueName: \"kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x\") pod \"d14428e4-2205-4a73-92f7-a2afbf0d9140\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " Oct 30 13:28:48.264487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264451 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache\") pod \"d14428e4-2205-4a73-92f7-a2afbf0d9140\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " Oct 30 13:28:48.264562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264507 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir\") pod \"d14428e4-2205-4a73-92f7-a2afbf0d9140\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " Oct 30 13:28:48.264642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264560 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs\" (UniqueName: \"kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs\") pod \"d14428e4-2205-4a73-92f7-a2afbf0d9140\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " Oct 30 13:28:48.264642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264609 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home\") pod \"d14428e4-2205-4a73-92f7-a2afbf0d9140\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " Oct 30 13:28:48.264787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264770 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm\") pod \"d14428e4-2205-4a73-92f7-a2afbf0d9140\" (UID: \"d14428e4-2205-4a73-92f7-a2afbf0d9140\") " Oct 30 13:28:48.264851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264811 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache" pod "d14428e4-2205-4a73-92f7-a2afbf0d9140" (UID: "d14428e4-2205-4a73-92f7-a2afbf0d9140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:48.264851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.264817 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir" pod "d14428e4-2205-4a73-92f7-a2afbf0d9140" (UID: "d14428e4-2205-4a73-92f7-a2afbf0d9140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:48.265062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.265039 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home" pod "d14428e4-2205-4a73-92f7-a2afbf0d9140" (UID: "d14428e4-2205-4a73-92f7-a2afbf0d9140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:48.265276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.265253 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.265349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.265274 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.265349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.265289 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.267237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.267211 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm" pod "d14428e4-2205-4a73-92f7-a2afbf0d9140" (UID: "d14428e4-2205-4a73-92f7-a2afbf0d9140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:48.267368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.267344 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x" pod "d14428e4-2205-4a73-92f7-a2afbf0d9140" (UID: "d14428e4-2205-4a73-92f7-a2afbf0d9140"). InnerVolumeSpecName "kube-api-access-dst2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:48.267662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.267633 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs" pod "d14428e4-2205-4a73-92f7-a2afbf0d9140" (UID: "d14428e4-2205-4a73-92f7-a2afbf0d9140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:48.367342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.367290 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14428e4-2205-4a73-92f7-a2afbf0d9140-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.367342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.367322 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dst2x\" (UniqueName: \"kubernetes.io/projected/d14428e4-2205-4a73-92f7-a2afbf0d9140-kube-api-access-dst2x\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.367342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.367337 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14428e4-2205-4a73-92f7-a2afbf0d9140-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.861164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.861125 5693 generic.go:356] "Generic (PLEG): container finished" podID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerID="922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372" exitCode=0 Oct 30 13:28:48.861346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.861180 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" Oct 30 13:28:48.861346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.861207 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" event={"ID":"d14428e4-2205-4a73-92f7-a2afbf0d9140","Type":"ContainerDied","Data":"922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372"} Oct 30 13:28:48.861346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.861246 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" event={"ID":"d14428e4-2205-4a73-92f7-a2afbf0d9140","Type":"ContainerDied","Data":"ced4fe7b4d6c27c927c28d96a7c09a8728cf9fe000f7cef829a4427334089d3d"} Oct 30 13:28:48.861346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.861262 5693 scope.go:122] "RemoveContainer" containerID="922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372" Oct 30 13:28:48.863142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.863121 5693 generic.go:356] "Generic (PLEG): container finished" podID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerID="d1637a72b19ea68ce7c23eab0ec3b15e085d50def8255a71f1e5b748b9a9e147" exitCode=0 Oct 30 13:28:48.863276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.863189 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" event={"ID":"d33dd336-737c-43a5-974f-ba026ceeaf0c","Type":"ContainerDied","Data":"d1637a72b19ea68ce7c23eab0ec3b15e085d50def8255a71f1e5b748b9a9e147"} Oct 30 13:28:48.863759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.863741 5693 scope.go:122] "RemoveContainer" containerID="d1637a72b19ea68ce7c23eab0ec3b15e085d50def8255a71f1e5b748b9a9e147" Oct 30 13:28:48.865305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.865273 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:28:48.865409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.865345 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:28:48.865409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.865383 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:28:48.875714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.875697 5693 scope.go:122] "RemoveContainer" containerID="922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372" Oct 30 13:28:48.875980 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:48.875957 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372\": container with ID starting with 922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372 not found: ID does not exist" containerID="922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372" Oct 30 13:28:48.876042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.875991 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372"} err="failed to get container status \"922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372\": rpc error: code = NotFound desc = could not find container \"922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372\": container with ID starting with 922744b619142da0536014f4289bbdfcd877b0c744362842b7d7963b4f60b372 not found: ID does not exist" Oct 30 13:28:48.911701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.911674 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh"] Oct 30 13:28:48.915031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:48.915011 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh"] Oct 30 13:28:49.117954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:49.117862 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5cee02/praxis-route-5cee02-kserve-5fc5dfcf57-92wsh" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": context deadline exceeded" Oct 30 13:28:49.872115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:49.872071 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" event={"ID":"d33dd336-737c-43a5-974f-ba026ceeaf0c","Type":"ContainerStarted","Data":"d06588f421559b82d02c1dbd1bf2ccd8415bc82829470b0d19e7f63460df772a"} Oct 30 13:28:49.967931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:49.967894 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:28:49.969300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:49.969270 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:28:49.969443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:49.969321 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:28:50.276624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:50.270945 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:50.306008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:50.298065 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" path="/var/lib/kubelet/pods/d14428e4-2205-4a73-92f7-a2afbf0d9140/volumes" Oct 30 13:28:50.967803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:50.967772 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:28:50.967998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:50.967829 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:28:51.452015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.451574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.603636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.603585 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.968622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.968053 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:28:51.968622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.968135 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.988311 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m"] Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990126 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990160 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990178 5693 cpu_manager.go:414] "containerMap: removing container" podUID="da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" containerName="maas-api" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990186 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" containerName="maas-api" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990200 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" containerName="payload-processing" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990207 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" containerName="payload-processing" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990231 5693 cpu_manager.go:414] "containerMap: removing container" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990238 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990263 5693 cpu_manager.go:414] "containerMap: removing container" podUID="31fa3123-b4f7-4edc-baa5-14d1f430741d" containerName="payload-pre-processing" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990270 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fa3123-b4f7-4edc-baa5-14d1f430741d" containerName="payload-pre-processing" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990283 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990290 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990384 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990393 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990409 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990418 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990619 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14428e4-2205-4a73-92f7-a2afbf0d9140" containerName="main" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990631 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e6d59-ddc6-4b7a-8df4-ec475f8dd796" containerName="maas-api" Oct 30 13:28:51.990623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990640 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7f62f9-35ea-40c7-91c6-a65fcd275cb3" containerName="payload-processing" Oct 30 13:28:51.992065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990651 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="860ef2b5-d62c-4059-90e9-e59a2019141c" containerName="istio-proxy" Oct 30 13:28:51.992065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990663 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fa3123-b4f7-4edc-baa5-14d1f430741d" containerName="payload-pre-processing" Oct 30 13:28:51.992065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990672 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" Oct 30 13:28:51.992065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990695 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc479868-520a-45f0-bc7c-ba06a8f95830" containerName="istio-proxy" Oct 30 13:28:51.992065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:51.990704 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e947291-0a68-48df-b65a-d3eeda038447" containerName="istio-proxy" Oct 30 13:28:52.003457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.003429 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.017170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.017044 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m"] Oct 30 13:28:52.032670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.025178 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-642765\"" Oct 30 13:28:52.066492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.065810 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-642765-dockercfg-8sxkc\"" Oct 30 13:28:52.066492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.066349 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:52.113357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.113322 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.113530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.113470 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtb2p\" (UniqueName: \"kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p\") pod \"payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.161897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.159987 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2"] Oct 30 13:28:52.171725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.171436 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.171725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.171466 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2"] Oct 30 13:28:52.208639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.208298 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:52.214694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.214664 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtb2p\" (UniqueName: \"kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p\") pod \"payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.214845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.214713 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.214910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.214879 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42vz\" (UniqueName: \"kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz\") pod \"payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.214991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.214977 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.215730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.215710 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.254545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.254509 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtb2p\" (UniqueName: \"kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p\") pod \"payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.316466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.316433 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42vz\" (UniqueName: \"kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz\") pod \"payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.316653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.316637 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.317362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.317343 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.354384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.354343 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42vz\" (UniqueName: \"kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz\") pod \"payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.367984 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.367947 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.491026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.490997 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:52.685789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.685763 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m"] Oct 30 13:28:52.688831 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:28:52.688799 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0f4b3f_655c_4d57_9cb4_c3ac8495c891.slice/crio-d445bfddcb933703d542ef9470ec9c5b07262bde2bc93c8e3b56ceb56ebe5710 WatchSource:0}: Error finding container d445bfddcb933703d542ef9470ec9c5b07262bde2bc93c8e3b56ceb56ebe5710: Status 404 returned error can't find the container with id d445bfddcb933703d542ef9470ec9c5b07262bde2bc93c8e3b56ceb56ebe5710 Oct 30 13:28:52.759673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.759087 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:52.841623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.836417 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2"] Oct 30 13:28:52.957510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.957473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" event={"ID":"0ba90414-5892-46f7-9dd3-266b8e5f87d2","Type":"ContainerStarted","Data":"85192898f16b6fc36b69e247ee3c86daf523309f65cdb57b55663cbdc4b7d3ce"} Oct 30 13:28:52.960171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.960135 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" event={"ID":"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891","Type":"ContainerStarted","Data":"fa01c7cd736fb4461965eedd6a9d2d432c682791745fcaa0085ac9289fec1d14"} Oct 30 13:28:52.960288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.960191 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" event={"ID":"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891","Type":"ContainerStarted","Data":"d445bfddcb933703d542ef9470ec9c5b07262bde2bc93c8e3b56ceb56ebe5710"} Oct 30 13:28:52.960288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.960215 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:28:52.968304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.967813 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:28:52.968304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:52.967865 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:28:53.002135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:53.002082 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" podStartSLOduration=2.002065592 podStartE2EDuration="2.002065592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:52.998673496 +0000 UTC m=+1508.865291668" watchObservedRunningTime="2026-10-30 13:28:53.002065592 +0000 UTC m=+1508.868683764" Oct 30 13:28:53.082449 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:28:53.082371 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:28:53.318292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:53.318259 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:28:53.967883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:53.967847 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:28:53.968327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:53.967905 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:28:53.968813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:53.968783 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" event={"ID":"0ba90414-5892-46f7-9dd3-266b8e5f87d2","Type":"ContainerStarted","Data":"76360496b95d2ab885b246b6044138fa361f175059d27c3bed542ae064c29635"} Oct 30 13:28:54.009010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.008953 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" podStartSLOduration=2.008940271 podStartE2EDuration="2.008940271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:54.006324064 +0000 UTC m=+1509.872942238" watchObservedRunningTime="2026-10-30 13:28:54.008940271 +0000 UTC m=+1509.875558452" Oct 30 13:28:54.144023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.135162 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.266683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.266587 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.717703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.711408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.968180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.968085 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:28:54.968180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.968156 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:28:54.977022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:54.976978 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:28:55.972856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:55.972824 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:28:55.974059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:55.974035 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:28:56.316005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:56.315965 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:56.766494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:56.766461 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:57.093166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.093086 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz"] Oct 30 13:28:57.098710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.098669 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.130570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.130526 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c684d2-openshift-default-dockercfg-f677f\"" Oct 30 13:28:57.134304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.134273 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz"] Oct 30 13:28:57.150196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150162 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150201 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150222 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150341 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150399 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150458 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150474 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrkb\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150500 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.150545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.150537 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.204322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.204296 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:57.251864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.251828 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.251876 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrkb\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.251917 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.251970 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252047 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252073 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252095 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252173 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252256 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252496 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.252980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.252952 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.253078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.253000 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.255152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.255125 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.255471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.255447 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.275614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.271785 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.275614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.272004 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrkb\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:57.359685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.357504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:57.439629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:57.431882 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:58.037624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:58.021638 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:28:58.037624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:58.021772 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:28:58.037624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:58.021828 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:28:58.037624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:58.022874 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz"] Oct 30 13:28:58.079725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:58.079526 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" event={"ID":"9480a4a3-e090-46a7-abb7-d9174cf8f1b9","Type":"ContainerStarted","Data":"0f3aa2ac1f6f487bce429fd28448f2aafbd85bf9b590e0a20f2f6bcdde97ee05"} Oct 30 13:28:58.567713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:58.559095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:59.100625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.099815 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:59.109703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.108573 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" event={"ID":"9480a4a3-e090-46a7-abb7-d9174cf8f1b9","Type":"ContainerStarted","Data":"9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2"} Oct 30 13:28:59.198623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.194077 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podStartSLOduration=2.194056382 podStartE2EDuration="2.194056382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:59.173742597 +0000 UTC m=+1515.040360773" watchObservedRunningTime="2026-10-30 13:28:59.194056382 +0000 UTC m=+1515.060674557" Oct 30 13:28:59.435688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.434134 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:28:59.435688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.434261 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:28:59.435688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.434307 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:28:59.521781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.521722 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:59.819741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.812215 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:59.843624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:28:59.840897 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:29:00.104682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:00.103696 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:29:00.104878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:00.104743 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:29:00.108619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:00.105198 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:29:00.118627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:00.118268 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:29:00.718617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:00.694337 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:29:00.804622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:00.804320 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 13:29:01.142514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:01.142479 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:29:01.143363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:01.143345 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:29:01.325040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:01.325009 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:01.619243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:01.619210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:01.777732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:01.777698 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.124217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.124180 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.269232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.269193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.414747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.414661 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw"] Oct 30 13:29:03.417630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.417571 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.419320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.419303 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.438070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.438042 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw"] Oct 30 13:29:03.446728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.446702 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c684d2\"" Oct 30 13:29:03.478998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.478966 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c684d2-dockercfg-ljhpk\"" Oct 30 13:29:03.596624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.557753 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.635181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.635150 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h25vh\" (UniqueName: \"kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh\") pod \"payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.646398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.613465 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct"] Oct 30 13:29:03.659989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.655582 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct"] Oct 30 13:29:03.669432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.663807 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.748022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.747974 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.748022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.748022 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h25vh\" (UniqueName: \"kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh\") pod \"payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.748269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.748180 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.748269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.748216 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gn8\" (UniqueName: \"kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8\") pod \"payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.748845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.748814 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.767718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.767691 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h25vh\" (UniqueName: \"kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh\") pod \"payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:03.849708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.849674 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gn8\" (UniqueName: \"kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8\") pod \"payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.849910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.849898 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.850556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.850537 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.869114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.869087 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gn8\" (UniqueName: \"kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8\") pod \"payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:03.971272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.971186 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:29:03.992419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:03.992387 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:04.034577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:04.034541 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:04.305399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:04.305369 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct"] Oct 30 13:29:04.309581 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:29:04.309552 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e006fa8_251a_4150_bad3_c7f47233d029.slice/crio-e649d9dbc368be7a5d6d910d72db07dd2346cd701f4d8052ec43d331350dfefa WatchSource:0}: Error finding container e649d9dbc368be7a5d6d910d72db07dd2346cd701f4d8052ec43d331350dfefa: Status 404 returned error can't find the container with id e649d9dbc368be7a5d6d910d72db07dd2346cd701f4d8052ec43d331350dfefa Oct 30 13:29:04.361964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:04.361936 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw"] Oct 30 13:29:04.363486 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:29:04.363454 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7022b194_fe9d_4290_832b_4380c59eec72.slice/crio-8dd5b79a738bac18ea024ddf6556fcd517c32270fea26d297469c63a124ce8db WatchSource:0}: Error finding container 8dd5b79a738bac18ea024ddf6556fcd517c32270fea26d297469c63a124ce8db: Status 404 returned error can't find the container with id 8dd5b79a738bac18ea024ddf6556fcd517c32270fea26d297469c63a124ce8db Oct 30 13:29:04.849166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:04.849117 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:05.214939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.214843 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" event={"ID":"4e006fa8-251a-4150-bad3-c7f47233d029","Type":"ContainerStarted","Data":"3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8"} Oct 30 13:29:05.214939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.214898 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" event={"ID":"4e006fa8-251a-4150-bad3-c7f47233d029","Type":"ContainerStarted","Data":"e649d9dbc368be7a5d6d910d72db07dd2346cd701f4d8052ec43d331350dfefa"} Oct 30 13:29:05.214939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.214926 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:05.216675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.216647 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" event={"ID":"7022b194-fe9d-4290-832b-4380c59eec72","Type":"ContainerStarted","Data":"81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf"} Oct 30 13:29:05.216802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.216705 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:05.216802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.216727 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" event={"ID":"7022b194-fe9d-4290-832b-4380c59eec72","Type":"ContainerStarted","Data":"8dd5b79a738bac18ea024ddf6556fcd517c32270fea26d297469c63a124ce8db"} Oct 30 13:29:05.254821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.254770 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" podStartSLOduration=2.254757783 podStartE2EDuration="2.254757783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:05.254105093 +0000 UTC m=+1521.120723283" watchObservedRunningTime="2026-10-30 13:29:05.254757783 +0000 UTC m=+1521.121375955" Oct 30 13:29:05.295239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.295162 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" podStartSLOduration=2.295143831 podStartE2EDuration="2.295143831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:05.291152925 +0000 UTC m=+1521.157771100" watchObservedRunningTime="2026-10-30 13:29:05.295143831 +0000 UTC m=+1521.161762007" Oct 30 13:29:05.985224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:05.985189 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:29:06.561279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:06.561247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:08.662512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:08.662472 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:09.557671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:09.557582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:10.911489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:10.911456 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.063033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:11.062996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.745587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:11.745549 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.891085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:11.891053 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:12.635071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:12.635030 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:14.285618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:14.281585 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:15.882409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:15.882328 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:16.083812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:16.083770 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:16.225942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:16.225914 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 13:29:16.226180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:16.226161 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 13:29:16.551839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:16.551734 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:16.719769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:16.719706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.580142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:17.580111 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.719002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:17.717525 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.865087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:17.865004 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:18.061024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:18.060992 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:18.209927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:18.209836 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:18.409521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:18.409473 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:20.094619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:20.094532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:20.974973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:20.941628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.282285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:21.282199 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.517759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:21.517718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.725407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:21.725376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:22.926626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:22.919276 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:23.063271 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:23.059813 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:29:23.135395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:23.135351 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:23.307089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:23.307047 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:24.870704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:24.870670 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:25.059663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:25.059440 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:26.711286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:26.711216 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:26.927784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:26.927748 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb"] Oct 30 13:29:26.928045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:26.928018 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" podUID="f20c80d5-5898-4508-9de3-d1d179dac629" containerName="payload-processing" containerID="cri-o://cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e" gracePeriod=30 Oct 30 13:29:26.958386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:26.958357 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq"] Oct 30 13:29:26.958657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:26.958632 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" podUID="95e797b2-9872-4e07-9995-395ff263cec3" containerName="payload-pre-processing" containerID="cri-o://8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916" gracePeriod=30 Oct 30 13:29:26.973065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:26.972966 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:27.119671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.118410 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:27.479046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.479012 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:29:27.482578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.482555 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:29:27.586503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.586467 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:27.589639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.589611 5693 generic.go:356] "Generic (PLEG): container finished" podID="f20c80d5-5898-4508-9de3-d1d179dac629" containerID="cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e" exitCode=0 Oct 30 13:29:27.589797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.589716 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" Oct 30 13:29:27.589797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.589755 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" event={"ID":"f20c80d5-5898-4508-9de3-d1d179dac629","Type":"ContainerDied","Data":"cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e"} Oct 30 13:29:27.589921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.589810 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb" event={"ID":"f20c80d5-5898-4508-9de3-d1d179dac629","Type":"ContainerDied","Data":"75b5faacbc7d030c2b4df678e8ddbb96ed159d9e98e28cde3e9450ec723fad40"} Oct 30 13:29:27.589921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.589845 5693 scope.go:122] "RemoveContainer" containerID="cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e" Oct 30 13:29:27.592021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.591995 5693 generic.go:356] "Generic (PLEG): container finished" podID="95e797b2-9872-4e07-9995-395ff263cec3" containerID="8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916" exitCode=0 Oct 30 13:29:27.592146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.592041 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" Oct 30 13:29:27.592146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.592117 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" event={"ID":"95e797b2-9872-4e07-9995-395ff263cec3","Type":"ContainerDied","Data":"8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916"} Oct 30 13:29:27.592242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.592158 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq" event={"ID":"95e797b2-9872-4e07-9995-395ff263cec3","Type":"ContainerDied","Data":"42bd23990cde6e31f079a93bdada748d0b16e9cb66f0a27780cb4106b475e670"} Oct 30 13:29:27.604703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.604670 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm\" (UniqueName: \"kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm\") pod \"95e797b2-9872-4e07-9995-395ff263cec3\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " Oct 30 13:29:27.604835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.604774 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume\") pod \"f20c80d5-5898-4508-9de3-d1d179dac629\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " Oct 30 13:29:27.604920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.604902 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume\") pod \"95e797b2-9872-4e07-9995-395ff263cec3\" (UID: \"95e797b2-9872-4e07-9995-395ff263cec3\") " Oct 30 13:29:27.605066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.605046 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4\" (UniqueName: \"kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4\") pod \"f20c80d5-5898-4508-9de3-d1d179dac629\" (UID: \"f20c80d5-5898-4508-9de3-d1d179dac629\") " Oct 30 13:29:27.609161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.609142 5693 scope.go:122] "RemoveContainer" containerID="cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e" Oct 30 13:29:27.609980 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:27.609952 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e\": container with ID starting with cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e not found: ID does not exist" containerID="cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e" Oct 30 13:29:27.610142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.610103 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e"} err="failed to get container status \"cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e\": rpc error: code = NotFound desc = could not find container \"cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e\": container with ID starting with cd4b736fce5352d18032a2291f4bdcda69f77673b29a93dc4c412f866f59420e not found: ID does not exist" Oct 30 13:29:27.610251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.610239 5693 scope.go:122] "RemoveContainer" containerID="8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916" Oct 30 13:29:27.611293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.611264 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume" pod "95e797b2-9872-4e07-9995-395ff263cec3" (UID: "95e797b2-9872-4e07-9995-395ff263cec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:27.611642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.611620 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4" pod "f20c80d5-5898-4508-9de3-d1d179dac629" (UID: "f20c80d5-5898-4508-9de3-d1d179dac629"). InnerVolumeSpecName "kube-api-access-hdxg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:27.612040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.611909 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume" pod "f20c80d5-5898-4508-9de3-d1d179dac629" (UID: "f20c80d5-5898-4508-9de3-d1d179dac629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:27.614736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.614710 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm" pod "95e797b2-9872-4e07-9995-395ff263cec3" (UID: "95e797b2-9872-4e07-9995-395ff263cec3"). InnerVolumeSpecName "kube-api-access-2j9sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:27.622522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.622501 5693 scope.go:122] "RemoveContainer" containerID="8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916" Oct 30 13:29:27.622842 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:27.622820 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916\": container with ID starting with 8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916 not found: ID does not exist" containerID="8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916" Oct 30 13:29:27.622917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.622849 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916"} err="failed to get container status \"8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916\": rpc error: code = NotFound desc = could not find container \"8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916\": container with ID starting with 8a063082ae20b0fd740ce04f9380b61d745ab65f35b2416eac7f91752d2d9916 not found: ID does not exist" Oct 30 13:29:27.710272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.710185 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e797b2-9872-4e07-9995-395ff263cec3-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:27.710272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.710216 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxg4\" (UniqueName: \"kubernetes.io/projected/f20c80d5-5898-4508-9de3-d1d179dac629-kube-api-access-hdxg4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:27.710272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.710226 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j9sm\" (UniqueName: \"kubernetes.io/projected/95e797b2-9872-4e07-9995-395ff263cec3-kube-api-access-2j9sm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:27.710272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.710236 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20c80d5-5898-4508-9de3-d1d179dac629-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:27.800675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.800634 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:27.982658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:27.978558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:28.015498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.015469 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq"] Oct 30 13:29:28.024060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.024030 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65e250-59c8895c99-cbpqq"] Oct 30 13:29:28.051959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.051927 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb"] Oct 30 13:29:28.054799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.054771 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-5d7fbfdd4-9q6wb"] Oct 30 13:29:28.248967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.248914 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:28.255628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.255573 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e797b2-9872-4e07-9995-395ff263cec3" path="/var/lib/kubelet/pods/95e797b2-9872-4e07-9995-395ff263cec3/volumes" Oct 30 13:29:28.256066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:28.256050 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20c80d5-5898-4508-9de3-d1d179dac629" path="/var/lib/kubelet/pods/f20c80d5-5898-4508-9de3-d1d179dac629/volumes" Oct 30 13:29:30.255623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:30.255012 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:31.269891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:31.269862 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:31.407540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:31.407507 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:31.908255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:31.908225 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:33.640621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:33.639234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:34.237569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:34.237531 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw"] Oct 30 13:29:34.397556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:34.397528 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:34.535770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:34.535688 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:34.810170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:34.810076 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:34.969511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:34.969475 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:35.100151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:35.100060 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp"] Oct 30 13:29:35.100677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:35.100641 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" containerID="cri-o://3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309" gracePeriod=30 Oct 30 13:29:35.165849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:35.165806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:35.352355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:35.342995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:37.175632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:37.174628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:38.543343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.543300 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq"] Oct 30 13:29:38.544351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.544325 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f20c80d5-5898-4508-9de3-d1d179dac629" containerName="payload-processing" Oct 30 13:29:38.544351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.544351 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20c80d5-5898-4508-9de3-d1d179dac629" containerName="payload-processing" Oct 30 13:29:38.544546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.544389 5693 cpu_manager.go:414] "containerMap: removing container" podUID="95e797b2-9872-4e07-9995-395ff263cec3" containerName="payload-pre-processing" Oct 30 13:29:38.544546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.544394 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e797b2-9872-4e07-9995-395ff263cec3" containerName="payload-pre-processing" Oct 30 13:29:38.544546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.544509 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e797b2-9872-4e07-9995-395ff263cec3" containerName="payload-pre-processing" Oct 30 13:29:38.544546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.544524 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20c80d5-5898-4508-9de3-d1d179dac629" containerName="payload-processing" Oct 30 13:29:38.548642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.548589 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.555784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.555759 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb"] Oct 30 13:29:38.560454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.560430 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.567453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.567426 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5852ff-openshift-default-dockercfg-76pkk\"" Oct 30 13:29:38.572564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.572538 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq"] Oct 30 13:29:38.601656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.599400 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb"] Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.742862 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.742912 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.742950 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.742990 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743030 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743093 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743123 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743147 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743224 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743289 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743325 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743361 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743403 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jvk\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743436 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743499 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743549 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.754638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.743586 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qbx\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859178 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859272 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859315 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859349 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859384 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jvk\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859428 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859547 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859612 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859647 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qbx\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859735 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859784 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859822 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859869 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.859915 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.860004 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.860036 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.860061 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.860105 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.860577 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.861669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.860911 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.870893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.869584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.870893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.869902 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.881622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.874659 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.881622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.875228 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.881622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.875736 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.893621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.884444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.893621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.884988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.893621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.885225 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.898820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.898761 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jvk\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.904422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.904396 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.910262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.910207 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.910262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.910235 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.910812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.910769 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:38.914366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.914319 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qbx\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.914892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.914841 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:38.917812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:38.917787 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:39.167830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.167746 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:39.179311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.179282 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:39.197488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.197447 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.242682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.241734 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" podUID="1e41bcb3-272d-486e-b0a8-26a713ce6e92" containerName="payload-processing" containerID="cri-o://a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4" gracePeriod=25 Oct 30 13:29:39.338177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.338137 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.518793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.491475 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.640668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.638895 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq"] Oct 30 13:29:39.655831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.654402 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:39.655831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.654490 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:39.655831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.654533 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:39.657023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.656991 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb"] Oct 30 13:29:39.686524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.686481 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:39.686771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.686748 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:39.686928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.686911 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:39.755932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.755902 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:29:39.804566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.801586 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr\" (UniqueName: \"kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr\") pod \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " Oct 30 13:29:39.804566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.801658 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume\") pod \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\" (UID: \"1e41bcb3-272d-486e-b0a8-26a713ce6e92\") " Oct 30 13:29:39.804566 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.802379 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume" pod "1e41bcb3-272d-486e-b0a8-26a713ce6e92" (UID: "1e41bcb3-272d-486e-b0a8-26a713ce6e92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:39.806435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.805012 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr" pod "1e41bcb3-272d-486e-b0a8-26a713ce6e92" (UID: "1e41bcb3-272d-486e-b0a8-26a713ce6e92"). InnerVolumeSpecName "kube-api-access-5vpdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:39.915782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.909087 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpdr\" (UniqueName: \"kubernetes.io/projected/1e41bcb3-272d-486e-b0a8-26a713ce6e92-kube-api-access-5vpdr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:39.915782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.909120 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e41bcb3-272d-486e-b0a8-26a713ce6e92-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:39.915782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.914164 5693 generic.go:356] "Generic (PLEG): container finished" podID="1e41bcb3-272d-486e-b0a8-26a713ce6e92" containerID="a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4" exitCode=0 Oct 30 13:29:39.915782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.914265 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" Oct 30 13:29:39.916212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.915855 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" event={"ID":"1e41bcb3-272d-486e-b0a8-26a713ce6e92","Type":"ContainerDied","Data":"a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4"} Oct 30 13:29:39.916212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.915932 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw" event={"ID":"1e41bcb3-272d-486e-b0a8-26a713ce6e92","Type":"ContainerDied","Data":"74d80f7adb20c75ee8720e73f1bab58ea579697129625c92561db959d489098a"} Oct 30 13:29:39.916212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.915964 5693 scope.go:122] "RemoveContainer" containerID="a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4" Oct 30 13:29:39.958675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.947449 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" event={"ID":"32d2f787-e433-41a5-ac92-43b3db7654a5","Type":"ContainerStarted","Data":"b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8"} Oct 30 13:29:39.958675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.947518 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" event={"ID":"32d2f787-e433-41a5-ac92-43b3db7654a5","Type":"ContainerStarted","Data":"1dc8a41a3170e6f70985149e60769234cd698332bc4344ad2b0d35eac5004dc6"} Oct 30 13:29:39.967621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.959668 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" event={"ID":"c3cd7236-a1ac-4031-ae4c-cee28ef0d476","Type":"ContainerStarted","Data":"e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a"} Oct 30 13:29:39.967621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:39.959742 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" event={"ID":"c3cd7236-a1ac-4031-ae4c-cee28ef0d476","Type":"ContainerStarted","Data":"f6ff169a63a02e2166d793d61bbc062f4491f5fd581a82ee3d9f8a78ba02e428"} Oct 30 13:29:40.022024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.004022 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podStartSLOduration=2.003999741 podStartE2EDuration="2.003999741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:39.994251367 +0000 UTC m=+1555.860869541" watchObservedRunningTime="2026-10-30 13:29:40.003999741 +0000 UTC m=+1555.870617915" Oct 30 13:29:40.082053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.081337 5693 scope.go:122] "RemoveContainer" containerID="a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4" Oct 30 13:29:40.082053 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:40.081897 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4\": container with ID starting with a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4 not found: ID does not exist" containerID="a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4" Oct 30 13:29:40.082053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.081955 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4"} err="failed to get container status \"a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4\": rpc error: code = NotFound desc = could not find container \"a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4\": container with ID starting with a1f9aa8ef92e622fcb79f8124b411e52d3a5d5bc8b8f6d7d74620c0ed5f169d4 not found: ID does not exist" Oct 30 13:29:40.111723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.111651 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podStartSLOduration=2.111625944 podStartE2EDuration="2.111625944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:40.071154981 +0000 UTC m=+1555.937773152" watchObservedRunningTime="2026-10-30 13:29:40.111625944 +0000 UTC m=+1555.978244154" Oct 30 13:29:40.175648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.172113 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:40.182746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.179953 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:40.183470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.182878 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:40.183470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.182929 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:40.183470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.183223 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:40.183470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.183266 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:40.198851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.190802 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw"] Oct 30 13:29:40.297234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.297200 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4091a5-6bffc47b58-g2lbw"] Oct 30 13:29:40.621063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.620965 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:40.677761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.677736 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:29:40.773623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.749502 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.774143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.773978 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.774443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.774424 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.774630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.774588 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.774785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.774770 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.774919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.774904 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.775101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.775084 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.775222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.775208 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.775423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.775405 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn\") pod \"7b065a33-cc37-42bd-b49b-cb1bff218923\" (UID: \"7b065a33-cc37-42bd-b49b-cb1bff218923\") " Oct 30 13:29:40.777213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.755344 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:40.777327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.777180 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:40.777561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.777535 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:40.778424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.778391 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:40.789867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.789808 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:40.794092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.792723 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:29:40.800906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.799699 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "kube-api-access-7btcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:40.892930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.813411 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.927979 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token" pod "7b065a33-cc37-42bd-b49b-cb1bff218923" (UID: "7b065a33-cc37-42bd-b49b-cb1bff218923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942558 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942614 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942629 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7btcn\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-kube-api-access-7btcn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942646 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942664 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b065a33-cc37-42bd-b49b-cb1bff218923-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942677 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942690 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942702 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b065a33-cc37-42bd-b49b-cb1bff218923-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:40.989621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:40.942720 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b065a33-cc37-42bd-b49b-cb1bff218923-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:41.054617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.052256 5693 generic.go:356] "Generic (PLEG): container finished" podID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerID="3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309" exitCode=0 Oct 30 13:29:41.110274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.079827 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" Oct 30 13:29:41.110274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.105665 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" event={"ID":"7b065a33-cc37-42bd-b49b-cb1bff218923","Type":"ContainerDied","Data":"3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309"} Oct 30 13:29:41.110274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.105834 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp" event={"ID":"7b065a33-cc37-42bd-b49b-cb1bff218923","Type":"ContainerDied","Data":"4c2a63a06f22f045098eefc3a86b7fe7e93f0dfedda2823bf7c30dc967c824b5"} Oct 30 13:29:41.110274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.106113 5693 scope.go:122] "RemoveContainer" containerID="3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309" Oct 30 13:29:41.219672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.217769 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:41.219672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.217852 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:41.305622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.301002 5693 scope.go:122] "RemoveContainer" containerID="e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6" Oct 30 13:29:41.305622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.302130 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:41.305622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.302196 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:41.414710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.414382 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp"] Oct 30 13:29:41.417258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.417212 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-65e250-openshift-default-b5c94c597-szgsp"] Oct 30 13:29:41.422644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.422593 5693 scope.go:122] "RemoveContainer" containerID="3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309" Oct 30 13:29:41.423079 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:41.423043 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309\": container with ID starting with 3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309 not found: ID does not exist" containerID="3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309" Oct 30 13:29:41.423175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.423091 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309"} err="failed to get container status \"3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309\": rpc error: code = NotFound desc = could not find container \"3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309\": container with ID starting with 3b96c1f5b6eb43cd6be2f90bb40a9c83f2183bfbc07fbe8a7f92e6c43d47e309 not found: ID does not exist" Oct 30 13:29:41.423175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.423123 5693 scope.go:122] "RemoveContainer" containerID="e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6" Oct 30 13:29:41.423443 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:41.423423 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6\": container with ID starting with e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6 not found: ID does not exist" containerID="e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6" Oct 30 13:29:41.423492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.423450 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6"} err="failed to get container status \"e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6\": rpc error: code = NotFound desc = could not find container \"e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6\": container with ID starting with e744098f1ef02de2abdde3dcc5aa6543ce188c516120dc61e9aac57cea8cbbe6 not found: ID does not exist" Oct 30 13:29:41.773624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:41.769514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:42.043225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.043137 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:42.180339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.170030 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:42.180339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.170112 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:42.184061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.184024 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:42.184362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.184332 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:42.291639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.291082 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e41bcb3-272d-486e-b0a8-26a713ce6e92" path="/var/lib/kubelet/pods/1e41bcb3-272d-486e-b0a8-26a713ce6e92/volumes" Oct 30 13:29:42.299856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.297997 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" path="/var/lib/kubelet/pods/7b065a33-cc37-42bd-b49b-cb1bff218923/volumes" Oct 30 13:29:42.480617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.480332 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:42.865620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.862286 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp"] Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.888789 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1e41bcb3-272d-486e-b0a8-26a713ce6e92" containerName="payload-processing" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.888821 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e41bcb3-272d-486e-b0a8-26a713ce6e92" containerName="payload-processing" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.888883 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.888891 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.888907 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.888915 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.889090 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.889106 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b065a33-cc37-42bd-b49b-cb1bff218923" containerName="istio-proxy" Oct 30 13:29:42.901623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.889124 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e41bcb3-272d-486e-b0a8-26a713ce6e92" containerName="payload-processing" Oct 30 13:29:42.925061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.924465 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp"] Oct 30 13:29:42.925061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:42.924630 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.076688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.030462 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077075 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077209 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077241 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnnxl\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077393 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077440 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077468 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077522 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.094658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.077570 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.169702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.126472 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.191930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.191900 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-22af91-openshift-default-dockercfg-g9dfk\"" Oct 30 13:29:43.245429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.245194 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.245429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.245325 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.245429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.245363 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnnxl\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.263621 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.263696 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.264731 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.264793 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.264824 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.264870 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.264901 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.265037 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.265510 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.265779 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.325761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.266532 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.348362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.335897 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.371624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.349495 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.371624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.349806 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.371624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.361221 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:43.371624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.361272 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:43.438002 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.429346 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:43.503634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.482198 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q"] Oct 30 13:29:43.503634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.490355 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" podUID="3e17f6d6-8a57-4aa1-b52d-d00e374449e8" containerName="maas-api" containerID="cri-o://04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f" gracePeriod=30 Oct 30 13:29:43.628619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.572520 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnnxl\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.649660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.636821 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.663622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.661813 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.753624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.738208 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:43.778773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:43.777133 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:44.169410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.168960 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:44.169410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.169022 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:44.205645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.185573 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:44.205645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.185651 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:44.867679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.810065 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:44.867679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.849743 5693 generic.go:356] "Generic (PLEG): container finished" podID="3e17f6d6-8a57-4aa1-b52d-d00e374449e8" containerID="04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f" exitCode=0 Oct 30 13:29:44.867679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:44.849820 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" event={"ID":"3e17f6d6-8a57-4aa1-b52d-d00e374449e8","Type":"ContainerDied","Data":"04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f"} Oct 30 13:29:45.095229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.093175 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:45.316625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.314075 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:45.316625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.314151 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:45.316625 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:45.314320 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e17f6d6_8a57_4aa1_b52d_d00e374449e8.slice/crio-04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e17f6d6_8a57_4aa1_b52d_d00e374449e8.slice/crio-conmon-04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:29:45.505639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.500250 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:45.519325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.519291 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp"] Oct 30 13:29:45.525623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.523635 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:45.525623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.523706 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:45.592659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.592059 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:45.592659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.592165 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:45.592659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.592212 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:45.718582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.718548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:45.734850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.734820 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:29:45.869338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.869246 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl\" (UniqueName: \"kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl\") pod \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " Oct 30 13:29:45.869642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.869622 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs\") pod \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " Oct 30 13:29:45.869786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.869772 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca\") pod \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " Oct 30 13:29:45.870001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.869983 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls\") pod \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\" (UID: \"3e17f6d6-8a57-4aa1-b52d-d00e374449e8\") " Oct 30 13:29:45.986130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:45.969666 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca" pod "3e17f6d6-8a57-4aa1-b52d-d00e374449e8" (UID: "3e17f6d6-8a57-4aa1-b52d-d00e374449e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:46.023256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.023228 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:46.112826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.103883 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls" pod "3e17f6d6-8a57-4aa1-b52d-d00e374449e8" (UID: "3e17f6d6-8a57-4aa1-b52d-d00e374449e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:46.112826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.104713 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs" pod "3e17f6d6-8a57-4aa1-b52d-d00e374449e8" (UID: "3e17f6d6-8a57-4aa1-b52d-d00e374449e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:46.112826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.112640 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl" pod "3e17f6d6-8a57-4aa1-b52d-d00e374449e8" (UID: "3e17f6d6-8a57-4aa1-b52d-d00e374449e8"). InnerVolumeSpecName "kube-api-access-6mqpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:46.121106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.121025 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" event={"ID":"cf039ca7-5fc7-4310-8c29-685bc70445d1","Type":"ContainerStarted","Data":"22410d6aab011d3a761413b421f44ccfdf1b1ca01c4c4d2ec3c1ff7ded646af3"} Oct 30 13:29:46.143627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.127621 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqpl\" (UniqueName: \"kubernetes.io/projected/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-kube-api-access-6mqpl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:46.166006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.165963 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:46.166273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.166253 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e17f6d6-8a57-4aa1-b52d-d00e374449e8-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:46.181657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.175740 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:46.181657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.175801 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:46.181657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.179049 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" event={"ID":"3e17f6d6-8a57-4aa1-b52d-d00e374449e8","Type":"ContainerDied","Data":"51c94cdf9bc5662e61d383e35551bb55cb5152b5e2faecf9f6deffe8fdb850fa"} Oct 30 13:29:46.181657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.179115 5693 scope.go:122] "RemoveContainer" containerID="04efe9e751208a43d6f4914660bf6b9e9ccef14ce67fe90cbb1971cec3bbf54f" Oct 30 13:29:46.181657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.179356 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q" Oct 30 13:29:46.186626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.185379 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:46.186626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.185430 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:46.324352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.324314 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q"] Oct 30 13:29:46.326555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:46.326529 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4091a5-57f4d89c9-jsf6q"] Oct 30 13:29:47.180668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.170039 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:47.180668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.170103 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:47.189759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.187730 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:47.189759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.187783 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:47.204141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.204106 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" event={"ID":"cf039ca7-5fc7-4310-8c29-685bc70445d1","Type":"ContainerStarted","Data":"30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4"} Oct 30 13:29:47.266622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.265689 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podStartSLOduration=5.265670124 podStartE2EDuration="5.265670124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:47.265105793 +0000 UTC m=+1563.131724005" watchObservedRunningTime="2026-10-30 13:29:47.265670124 +0000 UTC m=+1563.132288311" Oct 30 13:29:47.495621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.486447 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:47.744526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.744485 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:47.751433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:47.751352 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:48.168337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.168243 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 13:29:48.168337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.168309 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 13:29:48.179759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.179723 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:29:48.179899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.179779 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:29:48.217199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.217156 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:48.219581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.219552 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:29:48.260190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.260150 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e17f6d6-8a57-4aa1-b52d-d00e374449e8" path="/var/lib/kubelet/pods/3e17f6d6-8a57-4aa1-b52d-d00e374449e8/volumes" Oct 30 13:29:48.839710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.839653 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx"] Oct 30 13:29:48.840194 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.839959 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" podUID="d65836f0-a900-49e6-a446-d24d28d0c55e" containerName="maas-api" containerID="cri-o://1dc17deea1089ef6af877d7cb6fd775095e63a694956a4c70d99250c55aaa379" gracePeriod=30 Oct 30 13:29:48.898257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.898085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:48.953363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.953323 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m"] Oct 30 13:29:48.954545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.954087 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" podUID="1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" containerName="payload-processing" containerID="cri-o://fa01c7cd736fb4461965eedd6a9d2d432c682791745fcaa0085ac9289fec1d14" gracePeriod=30 Oct 30 13:29:48.975051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.975019 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2"] Oct 30 13:29:48.975267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:48.975246 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" podUID="0ba90414-5892-46f7-9dd3-266b8e5f87d2" containerName="payload-pre-processing" containerID="cri-o://76360496b95d2ab885b246b6044138fa361f175059d27c3bed542ae064c29635" gracePeriod=30 Oct 30 13:29:49.071149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.070984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.187547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.187502 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:49.189063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.188909 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:49.193271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.193228 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:29:49.205533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.203220 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:49.205533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.204489 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:49.207849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.207800 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:29:49.229530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.229428 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.236473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.236245 5693 generic.go:356] "Generic (PLEG): container finished" podID="0ba90414-5892-46f7-9dd3-266b8e5f87d2" containerID="76360496b95d2ab885b246b6044138fa361f175059d27c3bed542ae064c29635" exitCode=0 Oct 30 13:29:49.236473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.236390 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" event={"ID":"0ba90414-5892-46f7-9dd3-266b8e5f87d2","Type":"ContainerDied","Data":"76360496b95d2ab885b246b6044138fa361f175059d27c3bed542ae064c29635"} Oct 30 13:29:49.238837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.238713 5693 generic.go:356] "Generic (PLEG): container finished" podID="1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" containerID="fa01c7cd736fb4461965eedd6a9d2d432c682791745fcaa0085ac9289fec1d14" exitCode=0 Oct 30 13:29:49.238837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.238788 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" event={"ID":"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891","Type":"ContainerDied","Data":"fa01c7cd736fb4461965eedd6a9d2d432c682791745fcaa0085ac9289fec1d14"} Oct 30 13:29:49.244203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.244182 5693 generic.go:356] "Generic (PLEG): container finished" podID="d65836f0-a900-49e6-a446-d24d28d0c55e" containerID="1dc17deea1089ef6af877d7cb6fd775095e63a694956a4c70d99250c55aaa379" exitCode=0 Oct 30 13:29:49.244695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.244667 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" event={"ID":"d65836f0-a900-49e6-a446-d24d28d0c55e","Type":"ContainerDied","Data":"1dc17deea1089ef6af877d7cb6fd775095e63a694956a4c70d99250c55aaa379"} Oct 30 13:29:49.469501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.467192 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:29:49.561478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.561434 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs\") pod \"d65836f0-a900-49e6-a446-d24d28d0c55e\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " Oct 30 13:29:49.564753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.564728 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca\") pod \"d65836f0-a900-49e6-a446-d24d28d0c55e\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " Oct 30 13:29:49.564919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.564905 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls\") pod \"d65836f0-a900-49e6-a446-d24d28d0c55e\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " Oct 30 13:29:49.565145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.565110 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca" pod "d65836f0-a900-49e6-a446-d24d28d0c55e" (UID: "d65836f0-a900-49e6-a446-d24d28d0c55e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:49.565231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.565218 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg\" (UniqueName: \"kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg\") pod \"d65836f0-a900-49e6-a446-d24d28d0c55e\" (UID: \"d65836f0-a900-49e6-a446-d24d28d0c55e\") " Oct 30 13:29:49.566245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.566226 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65836f0-a900-49e6-a446-d24d28d0c55e-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.569470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.569424 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg" pod "d65836f0-a900-49e6-a446-d24d28d0c55e" (UID: "d65836f0-a900-49e6-a446-d24d28d0c55e"). InnerVolumeSpecName "kube-api-access-ppztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:49.570681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.570657 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls" pod "d65836f0-a900-49e6-a446-d24d28d0c55e" (UID: "d65836f0-a900-49e6-a446-d24d28d0c55e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:49.572049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.572017 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs" pod "d65836f0-a900-49e6-a446-d24d28d0c55e" (UID: "d65836f0-a900-49e6-a446-d24d28d0c55e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:49.650932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.650894 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:29:49.657907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.657309 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:29:49.668121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.668093 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppztg\" (UniqueName: \"kubernetes.io/projected/d65836f0-a900-49e6-a446-d24d28d0c55e-kube-api-access-ppztg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.668121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.668121 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.668298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.668130 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65836f0-a900-49e6-a446-d24d28d0c55e-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.768915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.768877 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume\") pod \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " Oct 30 13:29:49.769119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.769012 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p\" (UniqueName: \"kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p\") pod \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\" (UID: \"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891\") " Oct 30 13:29:49.769198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.769139 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz\" (UniqueName: \"kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz\") pod \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " Oct 30 13:29:49.769590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.769263 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume\") pod \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\" (UID: \"0ba90414-5892-46f7-9dd3-266b8e5f87d2\") " Oct 30 13:29:49.769590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.769337 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume" pod "1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" (UID: "1b0f4b3f-655c-4d57-9cb4-c3ac8495c891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:49.769990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.769965 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.770477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.770445 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume" pod "0ba90414-5892-46f7-9dd3-266b8e5f87d2" (UID: "0ba90414-5892-46f7-9dd3-266b8e5f87d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:49.771443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.771399 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.772007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.771985 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p" pod "1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" (UID: "1b0f4b3f-655c-4d57-9cb4-c3ac8495c891"). InnerVolumeSpecName "kube-api-access-qtb2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:49.772101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.772031 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz" pod "0ba90414-5892-46f7-9dd3-266b8e5f87d2" (UID: "0ba90414-5892-46f7-9dd3-266b8e5f87d2"). InnerVolumeSpecName "kube-api-access-c42vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:49.871678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.871569 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtb2p\" (UniqueName: \"kubernetes.io/projected/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891-kube-api-access-qtb2p\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.871678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.871609 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c42vz\" (UniqueName: \"kubernetes.io/projected/0ba90414-5892-46f7-9dd3-266b8e5f87d2-kube-api-access-c42vz\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.871678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.871624 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba90414-5892-46f7-9dd3-266b8e5f87d2-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.908056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:49.908025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:50.076914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.076878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:50.227679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.227645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:50.254091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.254056 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" Oct 30 13:29:50.255748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.255721 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" Oct 30 13:29:50.258030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.258005 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" Oct 30 13:29:50.258639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.258593 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2" event={"ID":"0ba90414-5892-46f7-9dd3-266b8e5f87d2","Type":"ContainerDied","Data":"85192898f16b6fc36b69e247ee3c86daf523309f65cdb57b55663cbdc4b7d3ce"} Oct 30 13:29:50.258746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.258654 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m" event={"ID":"1b0f4b3f-655c-4d57-9cb4-c3ac8495c891","Type":"ContainerDied","Data":"d445bfddcb933703d542ef9470ec9c5b07262bde2bc93c8e3b56ceb56ebe5710"} Oct 30 13:29:50.258746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.258672 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx" event={"ID":"d65836f0-a900-49e6-a446-d24d28d0c55e","Type":"ContainerDied","Data":"5a9188d412541ad939cdf874b0a2fe68ec58948b7ae96993e211dd335a458dda"} Oct 30 13:29:50.258746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.258694 5693 scope.go:122] "RemoveContainer" containerID="76360496b95d2ab885b246b6044138fa361f175059d27c3bed542ae064c29635" Oct 30 13:29:50.282813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.282791 5693 scope.go:122] "RemoveContainer" containerID="fa01c7cd736fb4461965eedd6a9d2d432c682791745fcaa0085ac9289fec1d14" Oct 30 13:29:50.295784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.295756 5693 scope.go:122] "RemoveContainer" containerID="1dc17deea1089ef6af877d7cb6fd775095e63a694956a4c70d99250c55aaa379" Oct 30 13:29:50.340519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.340454 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m"] Oct 30 13:29:50.344132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.344105 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-642765-57ffcc9488-rpt5m"] Oct 30 13:29:50.376656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.376610 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2"] Oct 30 13:29:50.381426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.381392 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-642765-75f7f754d7-brdl2"] Oct 30 13:29:50.382266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.382243 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:50.412974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.412943 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx"] Oct 30 13:29:50.417274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:50.417250 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-642765-5d8644f549-59tcx"] Oct 30 13:29:51.734306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.734277 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:51.802912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.802867 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt"] Oct 30 13:29:51.803552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803532 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3e17f6d6-8a57-4aa1-b52d-d00e374449e8" containerName="maas-api" Oct 30 13:29:51.803552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803549 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e17f6d6-8a57-4aa1-b52d-d00e374449e8" containerName="maas-api" Oct 30 13:29:51.803725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803569 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d65836f0-a900-49e6-a446-d24d28d0c55e" containerName="maas-api" Oct 30 13:29:51.803725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803574 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65836f0-a900-49e6-a446-d24d28d0c55e" containerName="maas-api" Oct 30 13:29:51.803725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803612 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" containerName="payload-processing" Oct 30 13:29:51.803725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803620 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" containerName="payload-processing" Oct 30 13:29:51.803725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803663 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0ba90414-5892-46f7-9dd3-266b8e5f87d2" containerName="payload-pre-processing" Oct 30 13:29:51.803725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803668 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba90414-5892-46f7-9dd3-266b8e5f87d2" containerName="payload-pre-processing" Oct 30 13:29:51.803917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803762 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba90414-5892-46f7-9dd3-266b8e5f87d2" containerName="payload-pre-processing" Oct 30 13:29:51.803917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803772 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e17f6d6-8a57-4aa1-b52d-d00e374449e8" containerName="maas-api" Oct 30 13:29:51.803917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803779 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65836f0-a900-49e6-a446-d24d28d0c55e" containerName="maas-api" Oct 30 13:29:51.803917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.803790 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" containerName="payload-processing" Oct 30 13:29:51.807425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.807405 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:51.825533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.825502 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt"] Oct 30 13:29:51.832921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.832894 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-22af91-dockercfg-wscfz\"" Oct 30 13:29:51.833502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.833478 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-22af91\"" Oct 30 13:29:51.859645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.858129 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl"] Oct 30 13:29:51.863895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.863871 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:51.878398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.878372 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl"] Oct 30 13:29:51.885885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.885860 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:51.904792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.904752 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szstf\" (UniqueName: \"kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf\") pod \"payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:51.905736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:51.905693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:52.058464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.056750 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.069899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.069857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:52.072038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.071795 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:52.072038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.071918 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.072228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.072191 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szstf\" (UniqueName: \"kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf\") pod \"payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:52.072340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.072289 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv88l\" (UniqueName: \"kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l\") pod \"payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.124619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.121266 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szstf\" (UniqueName: \"kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf\") pod \"payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:52.125705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.125412 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:52.174144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.174109 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv88l\" (UniqueName: \"kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l\") pod \"payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.174320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.174247 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.175007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.174986 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.193312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.193133 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv88l\" (UniqueName: \"kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l\") pod \"payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.258218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.258178 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba90414-5892-46f7-9dd3-266b8e5f87d2" path="/var/lib/kubelet/pods/0ba90414-5892-46f7-9dd3-266b8e5f87d2/volumes" Oct 30 13:29:52.258754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.258732 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0f4b3f-655c-4d57-9cb4-c3ac8495c891" path="/var/lib/kubelet/pods/1b0f4b3f-655c-4d57-9cb4-c3ac8495c891/volumes" Oct 30 13:29:52.259230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.259211 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65836f0-a900-49e6-a446-d24d28d0c55e" path="/var/lib/kubelet/pods/d65836f0-a900-49e6-a446-d24d28d0c55e/volumes" Oct 30 13:29:52.277161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.277136 5693 generic.go:356] "Generic (PLEG): container finished" podID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerID="9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2" exitCode=0 Oct 30 13:29:52.277252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.277205 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" event={"ID":"9480a4a3-e090-46a7-abb7-d9174cf8f1b9","Type":"ContainerDied","Data":"9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2"} Oct 30 13:29:52.277629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.277583 5693 scope.go:122] "RemoveContainer" containerID="9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2" Oct 30 13:29:52.279270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.279232 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:52.279363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.279326 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:52.279420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.279405 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:52.440032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.439985 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt"] Oct 30 13:29:52.447962 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:29:52.447925 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efcd45d_f7fe_49dc_b448_5d89f945a76c.slice/crio-d119f49a8c41743084a573d953a2e665101891dc5ca19823caae2771c88e9bd4 WatchSource:0}: Error finding container d119f49a8c41743084a573d953a2e665101891dc5ca19823caae2771c88e9bd4: Status 404 returned error can't find the container with id d119f49a8c41743084a573d953a2e665101891dc5ca19823caae2771c88e9bd4 Oct 30 13:29:52.479055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.479028 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:52.716482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.716408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.819958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.819926 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl"] Oct 30 13:29:52.825814 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:29:52.825779 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f6e0a0_cb2c_4a60_92f7_a5c6d621ae65.slice/crio-2574fda2431d3cf58e0a8a6d1a41b511b5dcc430f3a5afc749f2c00c3caa1ffd WatchSource:0}: Error finding container 2574fda2431d3cf58e0a8a6d1a41b511b5dcc430f3a5afc749f2c00c3caa1ffd: Status 404 returned error can't find the container with id 2574fda2431d3cf58e0a8a6d1a41b511b5dcc430f3a5afc749f2c00c3caa1ffd Oct 30 13:29:52.855903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.855879 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.985769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.985737 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz"] Oct 30 13:29:52.986060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:52.986020 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" containerID="cri-o://a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb" gracePeriod=30 Oct 30 13:29:53.056493 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:53.056456 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:29:53.058120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.058097 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:53.309012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.308910 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" event={"ID":"6efcd45d-f7fe-49dc-b448-5d89f945a76c","Type":"ContainerStarted","Data":"107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6"} Oct 30 13:29:53.309338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.309307 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" event={"ID":"6efcd45d-f7fe-49dc-b448-5d89f945a76c","Type":"ContainerStarted","Data":"d119f49a8c41743084a573d953a2e665101891dc5ca19823caae2771c88e9bd4"} Oct 30 13:29:53.310673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.310649 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:29:53.320676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.316057 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" event={"ID":"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65","Type":"ContainerStarted","Data":"62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8"} Oct 30 13:29:53.320676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.316120 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" event={"ID":"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65","Type":"ContainerStarted","Data":"2574fda2431d3cf58e0a8a6d1a41b511b5dcc430f3a5afc749f2c00c3caa1ffd"} Oct 30 13:29:53.320676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.316983 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:29:53.320676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.319527 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" event={"ID":"9480a4a3-e090-46a7-abb7-d9174cf8f1b9","Type":"ContainerStarted","Data":"c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d"} Oct 30 13:29:53.352052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.351971 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" podStartSLOduration=2.3519517 podStartE2EDuration="2.3519517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:53.350209461 +0000 UTC m=+1569.216827631" watchObservedRunningTime="2026-10-30 13:29:53.3519517 +0000 UTC m=+1569.218569873" Oct 30 13:29:53.387638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.387472 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" podStartSLOduration=2.387455727 podStartE2EDuration="2.387455727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:53.387231851 +0000 UTC m=+1569.253850031" watchObservedRunningTime="2026-10-30 13:29:53.387455727 +0000 UTC m=+1569.254073904" Oct 30 13:29:53.453633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.453272 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:29:53.453633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.453420 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:29:53.453633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.453467 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:29:53.737622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:53.732323 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:54.286898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:54.286866 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:54.451625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:54.449038 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:29:54.451625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:54.449101 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:29:55.446621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:55.437939 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:29:55.446621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:55.438013 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:29:55.604398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:55.604324 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:55.699153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:55.699060 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:29:55.699452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:55.699418 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:29:56.441022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:56.440981 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:29:56.444311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:56.441291 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:29:57.254622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.254146 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:57.438623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.434964 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:29:57.438623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.435040 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:29:57.478128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.478091 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:57.614438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.614347 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:57.766585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.766553 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz"] Oct 30 13:29:57.766900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.766866 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" containerID="cri-o://d06588f421559b82d02c1dbd1bf2ccd8415bc82829470b0d19e7f63460df772a" gracePeriod=30 Oct 30 13:29:57.848619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:57.843441 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:58.317140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.317112 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:29:58.381828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.381794 5693 generic.go:356] "Generic (PLEG): container finished" podID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerID="a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb" exitCode=0 Oct 30 13:29:58.382027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.381855 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" Oct 30 13:29:58.382027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.381966 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" event={"ID":"accafbd8-9c8c-4493-8954-d5a5813391e0","Type":"ContainerDied","Data":"a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb"} Oct 30 13:29:58.382027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.382018 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz" event={"ID":"accafbd8-9c8c-4493-8954-d5a5813391e0","Type":"ContainerDied","Data":"983bf65a556b506d7ecdfa256ecd30929b4924e2e3c3d9e5903d7bb95d4edcd5"} Oct 30 13:29:58.382196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.382067 5693 scope.go:122] "RemoveContainer" containerID="a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb" Oct 30 13:29:58.395184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.395160 5693 scope.go:122] "RemoveContainer" containerID="70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7" Oct 30 13:29:58.404728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.404699 5693 scope.go:122] "RemoveContainer" containerID="a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb" Oct 30 13:29:58.405015 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:58.404986 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb\": container with ID starting with a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb not found: ID does not exist" containerID="a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb" Oct 30 13:29:58.405170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.405022 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb"} err="failed to get container status \"a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb\": rpc error: code = NotFound desc = could not find container \"a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb\": container with ID starting with a761b6f6b69dd76a0914a3df1ab9f943d1d873c3a301d52812b4d14cd455accb not found: ID does not exist" Oct 30 13:29:58.405170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.405046 5693 scope.go:122] "RemoveContainer" containerID="70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7" Oct 30 13:29:58.405334 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:29:58.405302 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7\": container with ID starting with 70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7 not found: ID does not exist" containerID="70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7" Oct 30 13:29:58.405448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.405337 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7"} err="failed to get container status \"70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7\": rpc error: code = NotFound desc = could not find container \"70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7\": container with ID starting with 70075acac368f9a13d8c7a70b55f44185f44190b88473b921c14f1253255ccd7 not found: ID does not exist" Oct 30 13:29:58.426660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.426588 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.426803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.426735 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.426871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.426819 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.426871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.426856 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.426976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.426948 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.427036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.427015 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.427092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.427044 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.427145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.427100 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.427220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.427203 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token\") pod \"accafbd8-9c8c-4493-8954-d5a5813391e0\" (UID: \"accafbd8-9c8c-4493-8954-d5a5813391e0\") " Oct 30 13:29:58.428179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.428156 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:58.447719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.446845 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:58.449830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.448802 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:58.449830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.448883 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:29:58.449830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.449043 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:58.449830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.449385 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:29:58.449830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.449435 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:29:58.453482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.453433 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:58.457213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.457175 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "kube-api-access-twcgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:58.457325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.457219 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:58.458505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.458483 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy" pod "accafbd8-9c8c-4493-8954-d5a5813391e0" (UID: "accafbd8-9c8c-4493-8954-d5a5813391e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:58.538759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538666 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.538759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538702 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.538759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538718 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.538759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538732 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accafbd8-9c8c-4493-8954-d5a5813391e0-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.538759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538746 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.538759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538759 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.539354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538771 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accafbd8-9c8c-4493-8954-d5a5813391e0-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.539354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538786 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.539354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.538802 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcgj\" (UniqueName: \"kubernetes.io/projected/accafbd8-9c8c-4493-8954-d5a5813391e0-kube-api-access-twcgj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.752741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.752656 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz"] Oct 30 13:29:58.758063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:58.758026 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4091a5-openshift-default-d8b5d79c7-jnmbz"] Oct 30 13:29:59.118924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:59.118858 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:59.437844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:59.437765 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:29:59.438273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:59.438257 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:29:59.438980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:29:59.438960 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:30:00.146828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.146793 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4"] Oct 30 13:30:00.147464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.147448 5693 cpu_manager.go:414] "containerMap: removing container" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" Oct 30 13:30:00.147510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.147465 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" Oct 30 13:30:00.147572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.147563 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" Oct 30 13:30:00.155122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.155086 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6"] Oct 30 13:30:00.155319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.155295 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.156613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.156580 5693 cpu_manager.go:414] "containerMap: removing container" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" Oct 30 13:30:00.156752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.156684 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" Oct 30 13:30:00.158205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.158184 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" containerName="istio-proxy" Oct 30 13:30:00.164926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.164900 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.175406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.172064 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\"" Oct 30 13:30:00.177289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.177268 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd"] Oct 30 13:30:00.188518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.188494 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx"] Oct 30 13:30:00.189174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.189153 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.194876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.194856 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4"] Oct 30 13:30:00.195043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.195021 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6"] Oct 30 13:30:00.195043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.194984 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.201044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.201019 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx"] Oct 30 13:30:00.223006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.222971 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd"] Oct 30 13:30:00.255944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.255913 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accafbd8-9c8c-4493-8954-d5a5813391e0" path="/var/lib/kubelet/pods/accafbd8-9c8c-4493-8954-d5a5813391e0/volumes" Oct 30 13:30:00.283653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.283617 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.283836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.283707 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ft8t\" (UniqueName: \"kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.283836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.283734 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.283935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.283850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9jp\" (UniqueName: \"kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.283935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.283896 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.284009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.283981 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4d4\" (UniqueName: \"kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.284009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.284001 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmgt\" (UniqueName: \"kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.284154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.284142 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.336312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.336279 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr"] Oct 30 13:30:00.340669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.340647 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.359080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.357583 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr"] Oct 30 13:30:00.359578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.359549 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:30:00.385157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385124 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4d4\" (UniqueName: \"kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.385309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385158 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmgt\" (UniqueName: \"kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.385309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385295 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.385465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385452 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.385522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385504 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ft8t\" (UniqueName: \"kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.385578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385538 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.385746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385723 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9jp\" (UniqueName: \"kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.385841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.385771 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.386379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.386349 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.386537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.386508 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.387059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.387035 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.387286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.387269 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.404808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.404742 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4d4\" (UniqueName: \"kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.404808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.404795 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmgt\" (UniqueName: \"kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.405118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.405096 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9jp\" (UniqueName: \"kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.405184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.405156 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ft8t\" (UniqueName: \"kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.486607 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.486572 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.486968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.486701 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2bwr\" (UniqueName: \"kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.486968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.486844 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.493021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.492999 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:00.511362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.511338 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:00.528961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.528931 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:00.542962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.542926 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:00.593333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.591078 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.593333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.591147 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2bwr\" (UniqueName: \"kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.593333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.591261 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.593333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.592186 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.614625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.598091 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.770868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.770833 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2bwr\" (UniqueName: \"kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr\") pod \"maas-api-revoke-sub-e8d83bf2-xj8mr\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.958837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.958774 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:00.960047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:00.960022 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6"] Oct 30 13:30:00.962006 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:00.961978 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4409f36d_9c89_40a6_a21e_78b57d90948f.slice/crio-13f6ef545c181fbb6d4cb6a8d897c065da180329901480cb74f8f0b98e562c2f WatchSource:0}: Error finding container 13f6ef545c181fbb6d4cb6a8d897c065da180329901480cb74f8f0b98e562c2f: Status 404 returned error can't find the container with id 13f6ef545c181fbb6d4cb6a8d897c065da180329901480cb74f8f0b98e562c2f Oct 30 13:30:01.052821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.052635 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:01.058969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.058352 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4"] Oct 30 13:30:01.077620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.063637 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd"] Oct 30 13:30:01.093325 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:01.091828 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8638cd7c_6d5f_422d_a039_b702647577ee.slice/crio-a51596d03cf01f8ff48ef892f109388b56956ad09b9dd8f150e19ec47b81d2b7 WatchSource:0}: Error finding container a51596d03cf01f8ff48ef892f109388b56956ad09b9dd8f150e19ec47b81d2b7: Status 404 returned error can't find the container with id a51596d03cf01f8ff48ef892f109388b56956ad09b9dd8f150e19ec47b81d2b7 Oct 30 13:30:01.093889 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:01.093742 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b0d9f1_27cc_493a_ba06_070052833d54.slice/crio-0decc089f493bee3113663d5da5cb70b543adf60ad3bc6d740833c1bfc63554b WatchSource:0}: Error finding container 0decc089f493bee3113663d5da5cb70b543adf60ad3bc6d740833c1bfc63554b: Status 404 returned error can't find the container with id 0decc089f493bee3113663d5da5cb70b543adf60ad3bc6d740833c1bfc63554b Oct 30 13:30:01.137192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.137161 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx"] Oct 30 13:30:01.144867 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:01.140071 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4c0756_15f8_4eeb_8fb5_079a2f903d72.slice/crio-df8fe2349251bd6cdd220c23cdd11ece718ba12b938c41b6e983a4f10df4d6db WatchSource:0}: Error finding container df8fe2349251bd6cdd220c23cdd11ece718ba12b938c41b6e983a4f10df4d6db: Status 404 returned error can't find the container with id df8fe2349251bd6cdd220c23cdd11ece718ba12b938c41b6e983a4f10df4d6db Oct 30 13:30:01.336949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.336922 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr"] Oct 30 13:30:01.344052 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:01.344022 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b13c109_e844_4563_9824_6ddcf3728566.slice/crio-13f7c22bcac9bc57d25728d14a36a5f56362ed16216d5bc3c643afb07cd695bc WatchSource:0}: Error finding container 13f7c22bcac9bc57d25728d14a36a5f56362ed16216d5bc3c643afb07cd695bc: Status 404 returned error can't find the container with id 13f7c22bcac9bc57d25728d14a36a5f56362ed16216d5bc3c643afb07cd695bc Oct 30 13:30:01.406894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.406857 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerStarted","Data":"0decc089f493bee3113663d5da5cb70b543adf60ad3bc6d740833c1bfc63554b"} Oct 30 13:30:01.408269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.408236 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" event={"ID":"cd4c0756-15f8-4eeb-8fb5-079a2f903d72","Type":"ContainerStarted","Data":"df8fe2349251bd6cdd220c23cdd11ece718ba12b938c41b6e983a4f10df4d6db"} Oct 30 13:30:01.409684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.409661 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" event={"ID":"8638cd7c-6d5f-422d-a039-b702647577ee","Type":"ContainerStarted","Data":"a51596d03cf01f8ff48ef892f109388b56956ad09b9dd8f150e19ec47b81d2b7"} Oct 30 13:30:01.411107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.411084 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" event={"ID":"4409f36d-9c89-40a6-a21e-78b57d90948f","Type":"ContainerStarted","Data":"13f6ef545c181fbb6d4cb6a8d897c065da180329901480cb74f8f0b98e562c2f"} Oct 30 13:30:01.412324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:01.412302 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" event={"ID":"7b13c109-e844-4563-9824-6ddcf3728566","Type":"ContainerStarted","Data":"13f7c22bcac9bc57d25728d14a36a5f56362ed16216d5bc3c643afb07cd695bc"} Oct 30 13:30:02.420822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.420787 5693 generic.go:356] "Generic (PLEG): container finished" podID="7b13c109-e844-4563-9824-6ddcf3728566" containerID="9bb6f86505843e978a4061970dff830546ec2d890b7ef71b4bf9e609a091c487" exitCode=0 Oct 30 13:30:02.421222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.420874 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" event={"ID":"7b13c109-e844-4563-9824-6ddcf3728566","Type":"ContainerDied","Data":"9bb6f86505843e978a4061970dff830546ec2d890b7ef71b4bf9e609a091c487"} Oct 30 13:30:02.422415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.422394 5693 generic.go:356] "Generic (PLEG): container finished" podID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerID="765759fd854d8a8222802f8f2c4512dc0b04ab23d87dc4db721507d75ce6083b" exitCode=60 Oct 30 13:30:02.422507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.422473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerDied","Data":"765759fd854d8a8222802f8f2c4512dc0b04ab23d87dc4db721507d75ce6083b"} Oct 30 13:30:02.422796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.422776 5693 scope.go:122] "RemoveContainer" containerID="765759fd854d8a8222802f8f2c4512dc0b04ab23d87dc4db721507d75ce6083b" Oct 30 13:30:02.424245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.424216 5693 generic.go:356] "Generic (PLEG): container finished" podID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerID="528d07ac306778f33765448b90935fbddee878bff98e07b284a060dab94576f0" exitCode=60 Oct 30 13:30:02.424648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.424243 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" event={"ID":"cd4c0756-15f8-4eeb-8fb5-079a2f903d72","Type":"ContainerDied","Data":"528d07ac306778f33765448b90935fbddee878bff98e07b284a060dab94576f0"} Oct 30 13:30:02.424648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.424534 5693 scope.go:122] "RemoveContainer" containerID="528d07ac306778f33765448b90935fbddee878bff98e07b284a060dab94576f0" Oct 30 13:30:02.428440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.426341 5693 generic.go:356] "Generic (PLEG): container finished" podID="8638cd7c-6d5f-422d-a039-b702647577ee" containerID="124ac30847d545bf4941e83925178ef0b93da86908d9c4a52fa9871f7c9668dc" exitCode=60 Oct 30 13:30:02.428440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.426480 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" event={"ID":"8638cd7c-6d5f-422d-a039-b702647577ee","Type":"ContainerDied","Data":"124ac30847d545bf4941e83925178ef0b93da86908d9c4a52fa9871f7c9668dc"} Oct 30 13:30:02.428440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.426861 5693 scope.go:122] "RemoveContainer" containerID="124ac30847d545bf4941e83925178ef0b93da86908d9c4a52fa9871f7c9668dc" Oct 30 13:30:02.428997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.428975 5693 generic.go:356] "Generic (PLEG): container finished" podID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerID="1955fb0ceda11b3921ed483825599b553e247f58c04557fe75dc16eed949e090" exitCode=60 Oct 30 13:30:02.429077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.429029 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" event={"ID":"4409f36d-9c89-40a6-a21e-78b57d90948f","Type":"ContainerDied","Data":"1955fb0ceda11b3921ed483825599b553e247f58c04557fe75dc16eed949e090"} Oct 30 13:30:02.429241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.429228 5693 scope.go:122] "RemoveContainer" containerID="1955fb0ceda11b3921ed483825599b553e247f58c04557fe75dc16eed949e090" Oct 30 13:30:02.835696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:02.834850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:03.454831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.454796 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" event={"ID":"cd4c0756-15f8-4eeb-8fb5-079a2f903d72","Type":"ContainerStarted","Data":"ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f"} Oct 30 13:30:03.459448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.459372 5693 generic.go:356] "Generic (PLEG): container finished" podID="8638cd7c-6d5f-422d-a039-b702647577ee" containerID="8e77f8ca3ee27015b0618541a232cf6a9c6a8d735b0283a0b461b777b2189d49" exitCode=60 Oct 30 13:30:03.459568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.459549 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" event={"ID":"8638cd7c-6d5f-422d-a039-b702647577ee","Type":"ContainerDied","Data":"8e77f8ca3ee27015b0618541a232cf6a9c6a8d735b0283a0b461b777b2189d49"} Oct 30 13:30:03.459652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.459641 5693 scope.go:122] "RemoveContainer" containerID="124ac30847d545bf4941e83925178ef0b93da86908d9c4a52fa9871f7c9668dc" Oct 30 13:30:03.460047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.460029 5693 scope.go:122] "RemoveContainer" containerID="8e77f8ca3ee27015b0618541a232cf6a9c6a8d735b0283a0b461b777b2189d49" Oct 30 13:30:03.460379 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:03.460360 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4_odh-ai-gateway-infra(8638cd7c-6d5f-422d-a039-b702647577ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" Oct 30 13:30:03.468752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.467773 5693 generic.go:356] "Generic (PLEG): container finished" podID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerID="d06588f421559b82d02c1dbd1bf2ccd8415bc82829470b0d19e7f63460df772a" exitCode=0 Oct 30 13:30:03.468752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.467873 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" event={"ID":"d33dd336-737c-43a5-974f-ba026ceeaf0c","Type":"ContainerDied","Data":"d06588f421559b82d02c1dbd1bf2ccd8415bc82829470b0d19e7f63460df772a"} Oct 30 13:30:03.480227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.480203 5693 generic.go:356] "Generic (PLEG): container finished" podID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerID="ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f" exitCode=60 Oct 30 13:30:03.480390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.480370 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" event={"ID":"4409f36d-9c89-40a6-a21e-78b57d90948f","Type":"ContainerDied","Data":"ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f"} Oct 30 13:30:03.481162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.480733 5693 scope.go:122] "RemoveContainer" containerID="ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f" Oct 30 13:30:03.481162 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:03.480978 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6_odh-ai-gateway-infra(4409f36d-9c89-40a6-a21e-78b57d90948f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" Oct 30 13:30:03.483813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.483784 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerStarted","Data":"58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f"} Oct 30 13:30:03.484178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.484152 5693 scope.go:122] "RemoveContainer" containerID="58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f" Oct 30 13:30:03.484535 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:03.484440 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd_odh-ai-gateway-infra(a9b0d9f1-27cc-493a-ba06-070052833d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" Oct 30 13:30:03.507406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.507337 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" podStartSLOduration=3.507318228 podStartE2EDuration="3.507318228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.504239785 +0000 UTC m=+1579.370857959" watchObservedRunningTime="2026-10-30 13:30:03.507318228 +0000 UTC m=+1579.373936405" Oct 30 13:30:03.518417 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:03.518376 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b0d9f1_27cc_493a_ba06_070052833d54.slice/crio-58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4409f36d_9c89_40a6_a21e_78b57d90948f.slice/crio-conmon-ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:30:03.639832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.639797 5693 scope.go:122] "RemoveContainer" containerID="d1637a72b19ea68ce7c23eab0ec3b15e085d50def8255a71f1e5b748b9a9e147" Oct 30 13:30:03.648649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.648626 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:30:03.672397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.672379 5693 scope.go:122] "RemoveContainer" containerID="1955fb0ceda11b3921ed483825599b553e247f58c04557fe75dc16eed949e090" Oct 30 13:30:03.675348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.675330 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:03.748225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.748196 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.748386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.748280 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.748386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.748327 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca\") pod \"7b13c109-e844-4563-9824-6ddcf3728566\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " Oct 30 13:30:03.748585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.748556 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:03.748794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.748759 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca" pod "7b13c109-e844-4563-9824-6ddcf3728566" (UID: "7b13c109-e844-4563-9824-6ddcf3728566"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:03.749274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749257 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749292 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749335 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr\" (UniqueName: \"kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr\") pod \"7b13c109-e844-4563-9824-6ddcf3728566\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " Oct 30 13:30:03.749447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749383 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749427 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749471 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749529 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749628 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl\") pod \"d33dd336-737c-43a5-974f-ba026ceeaf0c\" (UID: \"d33dd336-737c-43a5-974f-ba026ceeaf0c\") " Oct 30 13:30:03.749704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749648 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:03.749704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.749687 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca\") pod \"7b13c109-e844-4563-9824-6ddcf3728566\" (UID: \"7b13c109-e844-4563-9824-6ddcf3728566\") " Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750146 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750450 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750773 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750795 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-openshift-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750812 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750827 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.750876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.750840 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.751759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.751739 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:03.752097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.752071 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:03.752301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.752275 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:30:03.752487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.752459 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:03.754405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.754384 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca" pod "7b13c109-e844-4563-9824-6ddcf3728566" (UID: "7b13c109-e844-4563-9824-6ddcf3728566"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:03.754578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.754555 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr" pod "7b13c109-e844-4563-9824-6ddcf3728566" (UID: "7b13c109-e844-4563-9824-6ddcf3728566"). InnerVolumeSpecName "kube-api-access-w2bwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:03.754845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.754819 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl" pod "d33dd336-737c-43a5-974f-ba026ceeaf0c" (UID: "d33dd336-737c-43a5-974f-ba026ceeaf0c"). InnerVolumeSpecName "kube-api-access-w4qxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852049 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33dd336-737c-43a5-974f-ba026ceeaf0c-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852078 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2bwr\" (UniqueName: \"kubernetes.io/projected/7b13c109-e844-4563-9824-6ddcf3728566-kube-api-access-w2bwr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852088 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852098 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852106 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4qxl\" (UniqueName: \"kubernetes.io/projected/d33dd336-737c-43a5-974f-ba026ceeaf0c-kube-api-access-w4qxl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852114 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b13c109-e844-4563-9824-6ddcf3728566-maas-api-xks-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.852132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:03.852122 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33dd336-737c-43a5-974f-ba026ceeaf0c-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:04.492680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.492644 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" event={"ID":"d33dd336-737c-43a5-974f-ba026ceeaf0c","Type":"ContainerDied","Data":"2d93d235cff5f6bc230b8c8ce4753c5058cc9649d45e1a6fb00c297833e75126"} Oct 30 13:30:04.493167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.492691 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz" Oct 30 13:30:04.493167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.492704 5693 scope.go:122] "RemoveContainer" containerID="d06588f421559b82d02c1dbd1bf2ccd8415bc82829470b0d19e7f63460df772a" Oct 30 13:30:04.495011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.494991 5693 scope.go:122] "RemoveContainer" containerID="ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f" Oct 30 13:30:04.495467 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:04.495443 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6_odh-ai-gateway-infra(4409f36d-9c89-40a6-a21e-78b57d90948f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" Oct 30 13:30:04.496246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.496230 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" Oct 30 13:30:04.496354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.496253 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr" event={"ID":"7b13c109-e844-4563-9824-6ddcf3728566","Type":"ContainerDied","Data":"13f7c22bcac9bc57d25728d14a36a5f56362ed16216d5bc3c643afb07cd695bc"} Oct 30 13:30:04.496354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.496288 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13f7c22bcac9bc57d25728d14a36a5f56362ed16216d5bc3c643afb07cd695bc" Oct 30 13:30:04.498345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.498324 5693 generic.go:356] "Generic (PLEG): container finished" podID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerID="58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f" exitCode=60 Oct 30 13:30:04.498433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.498411 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerDied","Data":"58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f"} Oct 30 13:30:04.498845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.498826 5693 scope.go:122] "RemoveContainer" containerID="58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f" Oct 30 13:30:04.499114 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:04.499088 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd_odh-ai-gateway-infra(a9b0d9f1-27cc-493a-ba06-070052833d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" Oct 30 13:30:04.500534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.500512 5693 generic.go:356] "Generic (PLEG): container finished" podID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerID="ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f" exitCode=60 Oct 30 13:30:04.500654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.500575 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" event={"ID":"cd4c0756-15f8-4eeb-8fb5-079a2f903d72","Type":"ContainerDied","Data":"ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f"} Oct 30 13:30:04.500828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.500814 5693 scope.go:122] "RemoveContainer" containerID="ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f" Oct 30 13:30:04.501046 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:04.501023 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx_odh-ai-gateway-infra(cd4c0756-15f8-4eeb-8fb5-079a2f903d72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" Oct 30 13:30:04.502336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.502321 5693 scope.go:122] "RemoveContainer" containerID="8e77f8ca3ee27015b0618541a232cf6a9c6a8d735b0283a0b461b777b2189d49" Oct 30 13:30:04.502505 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:04.502489 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4_odh-ai-gateway-infra(8638cd7c-6d5f-422d-a039-b702647577ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" Oct 30 13:30:04.505092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.505075 5693 scope.go:122] "RemoveContainer" containerID="765759fd854d8a8222802f8f2c4512dc0b04ab23d87dc4db721507d75ce6083b" Oct 30 13:30:04.519179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.519156 5693 scope.go:122] "RemoveContainer" containerID="528d07ac306778f33765448b90935fbddee878bff98e07b284a060dab94576f0" Oct 30 13:30:04.621828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.621796 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz"] Oct 30 13:30:04.626443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.626416 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-642765-openshift-default-5c5cc5bf89-xdmnz"] Oct 30 13:30:04.711665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:04.700850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:05.359753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:05.359723 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 13:30:05.359918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:05.359852 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 13:30:05.462255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:05.462214 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:05.513056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:05.513028 5693 scope.go:122] "RemoveContainer" containerID="58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f" Oct 30 13:30:05.513481 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:05.513372 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd_odh-ai-gateway-infra(a9b0d9f1-27cc-493a-ba06-070052833d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" Oct 30 13:30:05.514549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:05.514528 5693 scope.go:122] "RemoveContainer" containerID="ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f" Oct 30 13:30:05.514825 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:05.514806 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx_odh-ai-gateway-infra(cd4c0756-15f8-4eeb-8fb5-079a2f903d72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" Oct 30 13:30:05.902489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:05.902454 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.257298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.257257 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" path="/var/lib/kubelet/pods/d33dd336-737c-43a5-974f-ba026ceeaf0c/volumes" Oct 30 13:30:06.308733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.308696 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js"] Oct 30 13:30:06.309336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309318 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" Oct 30 13:30:06.309336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309335 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" Oct 30 13:30:06.309492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309380 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7b13c109-e844-4563-9824-6ddcf3728566" containerName="revoke-keys" Oct 30 13:30:06.309492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309385 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b13c109-e844-4563-9824-6ddcf3728566" containerName="revoke-keys" Oct 30 13:30:06.309492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309471 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b13c109-e844-4563-9824-6ddcf3728566" containerName="revoke-keys" Oct 30 13:30:06.309492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309481 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" Oct 30 13:30:06.309492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.309488 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" Oct 30 13:30:06.314675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.314651 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.339343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.339316 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js"] Oct 30 13:30:06.345218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.345193 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-552cc3-openshift-default-dockercfg-4r4fk\"" Oct 30 13:30:06.391961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.391750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.391961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.391798 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.391961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.391853 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.391961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.391901 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.392244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.392138 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjfp9\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.392244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.392217 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.392424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.392405 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.392587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.392568 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.392681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.392635 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.406196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.406172 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb"] Oct 30 13:30:06.406867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.406852 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" Oct 30 13:30:06.406929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.406867 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33dd336-737c-43a5-974f-ba026ceeaf0c" containerName="istio-proxy" Oct 30 13:30:06.410553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.410537 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.419658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.419632 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.431992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.431969 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb"] Oct 30 13:30:06.437230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.437199 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-22af91-serving-cert\"" Oct 30 13:30:06.493902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.493863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.493919 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.493969 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.494069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.493996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjfp9\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494027 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494115 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.494276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494144 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.494388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494332 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494451 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhs2\" (UniqueName: \"kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.494529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494481 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494498 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494499 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494550 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494699 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494851 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.494948 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494923 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.495005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.494953 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.495320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.495295 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.496818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.496797 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.496927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.496913 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.513421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.513367 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.514014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.513995 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjfp9\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9\") pod \"e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.606627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.600636 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhs2\" (UniqueName: \"kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.606627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.600785 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.606627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.600832 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.606627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.600859 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.606627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.601727 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.663367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.648144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.663367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.659318 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.663367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.660357 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:06.698660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.693235 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.698660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.696886 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhs2\" (UniqueName: \"kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2\") pod \"maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.748621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.727961 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:06.828967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.828927 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.992914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:06.992862 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:07.104014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.103927 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb"] Oct 30 13:30:07.106774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.106746 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js"] Oct 30 13:30:07.119652 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:07.119625 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0869e_b1e7_44a9_bde1_507f205eb920.slice/crio-aed02a48cdaef8f38deb590d3ce2f12706bae22958b6e1cc3204b09429e9076d WatchSource:0}: Error finding container aed02a48cdaef8f38deb590d3ce2f12706bae22958b6e1cc3204b09429e9076d: Status 404 returned error can't find the container with id aed02a48cdaef8f38deb590d3ce2f12706bae22958b6e1cc3204b09429e9076d Oct 30 13:30:07.120379 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:07.120355 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a928a3_4554_425d_a2e5_38a118e6729e.slice/crio-4d0f64299c3cfc39647507ae61adf21b13bdcc4a38445059ba39d025c43a2340 WatchSource:0}: Error finding container 4d0f64299c3cfc39647507ae61adf21b13bdcc4a38445059ba39d025c43a2340: Status 404 returned error can't find the container with id 4d0f64299c3cfc39647507ae61adf21b13bdcc4a38445059ba39d025c43a2340 Oct 30 13:30:07.123206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.123173 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:07.123283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.123244 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:07.123283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.123274 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:07.327325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.327294 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd"] Oct 30 13:30:07.331993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.331965 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.347634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.347589 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-f12b80-openshift-default-dockercfg-p4bvb\"" Oct 30 13:30:07.366518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.366439 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd"] Oct 30 13:30:07.416640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.416588 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.416811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.416661 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.416811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.416724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.416904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.416845 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.417004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.416977 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.417124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.417106 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.417185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.417135 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.417280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.417247 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.417333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.417302 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrt4\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.454880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.453869 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.519527 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.519671 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.519771 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.519804 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.519904 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.519938 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrt4\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.520006 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.520044 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.520076 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.520943 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.524620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.521247 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.538830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.537262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.546629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.541786 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.605628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.603411 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.623738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.615890 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" event={"ID":"34a928a3-4554-425d-a2e5-38a118e6729e","Type":"ContainerStarted","Data":"0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591"} Oct 30 13:30:07.623738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.615977 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" event={"ID":"34a928a3-4554-425d-a2e5-38a118e6729e","Type":"ContainerStarted","Data":"4d0f64299c3cfc39647507ae61adf21b13bdcc4a38445059ba39d025c43a2340"} Oct 30 13:30:07.623738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.622586 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.633631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.630847 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.633631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.631538 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrt4\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.642641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.636550 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" event={"ID":"15a0869e-b1e7-44a9-bde1-507f205eb920","Type":"ContainerStarted","Data":"aed02a48cdaef8f38deb590d3ce2f12706bae22958b6e1cc3204b09429e9076d"} Oct 30 13:30:07.674620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.668702 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:07.675056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.675025 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo\") pod \"e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:07.724960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.724890 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podStartSLOduration=1.7248693899999998 podStartE2EDuration="1.72486939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:07.718012221 +0000 UTC m=+1583.584630420" watchObservedRunningTime="2026-10-30 13:30:07.72486939 +0000 UTC m=+1583.591487564" Oct 30 13:30:07.735627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.732820 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:07.952456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:07.952369 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:08.275730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.275699 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd"] Oct 30 13:30:08.279151 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:08.279118 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd687829a_5d4b_469f_b1a3_262f612892a0.slice/crio-2ce1e7be15f023a2e73904e45a93f53ea43351139f616409f6170f86bf43ee92 WatchSource:0}: Error finding container 2ce1e7be15f023a2e73904e45a93f53ea43351139f616409f6170f86bf43ee92: Status 404 returned error can't find the container with id 2ce1e7be15f023a2e73904e45a93f53ea43351139f616409f6170f86bf43ee92 Oct 30 13:30:08.282715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.282676 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:08.282799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.282751 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:08.282799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.282777 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:08.558689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.553069 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:08.644886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.644843 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" event={"ID":"d687829a-5d4b-469f-b1a3-262f612892a0","Type":"ContainerStarted","Data":"b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737"} Oct 30 13:30:08.645056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.644916 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" event={"ID":"d687829a-5d4b-469f-b1a3-262f612892a0","Type":"ContainerStarted","Data":"2ce1e7be15f023a2e73904e45a93f53ea43351139f616409f6170f86bf43ee92"} Oct 30 13:30:08.646578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.646547 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" event={"ID":"15a0869e-b1e7-44a9-bde1-507f205eb920","Type":"ContainerStarted","Data":"2c0d50c44a5714850907fe82fd1d5cf00ae2c4a18f16b52669fdac22cef5f900"} Oct 30 13:30:08.646866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.646846 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:08.671484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.671457 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:08.701586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.701529 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podStartSLOduration=1.701513559 podStartE2EDuration="1.701513559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:08.698522552 +0000 UTC m=+1584.565140726" watchObservedRunningTime="2026-10-30 13:30:08.701513559 +0000 UTC m=+1584.568131732" Oct 30 13:30:08.906327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.906198 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" podStartSLOduration=2.575964474 podStartE2EDuration="2.906176788s" totalImagesPullingTime="330.212314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:08.900932957 +0000 UTC m=+1584.767551133" watchObservedRunningTime="2026-10-30 13:30:08.906176788 +0000 UTC m=+1584.772794962" Oct 30 13:30:08.952967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:08.952932 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:09.653578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:09.653544 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:09.654369 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:09.654348 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:09.959707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:09.959632 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:10.297547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:10.297505 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:10.614611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:10.614510 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:10.660287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:10.660252 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:10.661307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:10.661285 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:30:10.945222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:10.945127 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:11.322563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:11.322533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:11.581377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:11.581295 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.310585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.310552 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.492045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.492018 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.603778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.603701 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7"] Oct 30 13:30:12.609673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.609651 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.636036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.635991 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7"] Oct 30 13:30:12.643697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.643668 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-552cc3\"" Oct 30 13:30:12.643922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.643831 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-552cc3-dockercfg-r5r6q\"" Oct 30 13:30:12.654586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.654557 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d"] Oct 30 13:30:12.660175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.660156 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.701851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.701791 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d"] Oct 30 13:30:12.704935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.704912 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn"] Oct 30 13:30:12.709460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.709440 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:12.728057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728022 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728128 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728185 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728249 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwkq\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728281 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728317 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.728468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728453 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728485 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728561 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.728696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.728653 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dbd\" (UniqueName: \"kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd\") pod \"payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.731029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.731006 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn"] Oct 30 13:30:12.749066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.749035 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f45f04-openshift-default-dockercfg-vzt9g\"" Oct 30 13:30:12.750237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.750213 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.849673 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.849758 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.849797 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.849827 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwkq\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.849863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.849902 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850001 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850091 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850169 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dbd\" (UniqueName: \"kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd\") pod \"payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850236 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850307 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.850375 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n68r\" (UniqueName: \"kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r\") pod \"payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.851053 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.864744 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.865330 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.866076 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.880141 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.881026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.880409 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.929683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.885387 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.929683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.892950 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.929683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.903275 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwkq\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.966618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.939365 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dbd\" (UniqueName: \"kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd\") pod \"payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:12.966618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.963334 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token\") pod \"e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:12.966863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.966662 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:12.966863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.966794 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n68r\" (UniqueName: \"kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r\") pod \"payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:12.975109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.968175 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:13.017645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.975841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.017645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:12.984059 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:13.047619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.043315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n68r\" (UniqueName: \"kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r\") pod \"payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:13.161586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.161501 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.236622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.226692 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:13.365758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.357959 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:13.459748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.459162 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.468456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.468389 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d"] Oct 30 13:30:13.474920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.474374 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:13.474920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.474461 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:13.474920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.474506 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:13.686423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.686388 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq"] Oct 30 13:30:13.691089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.691065 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.714519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.714263 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" event={"ID":"61a4c160-ebaf-40a4-a5b5-f508d2aaba50","Type":"ContainerStarted","Data":"d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a"} Oct 30 13:30:13.715032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.714806 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" event={"ID":"61a4c160-ebaf-40a4-a5b5-f508d2aaba50","Type":"ContainerStarted","Data":"5b7e909fd5212afdb0937a7bfced0706e1ed135fc875e3244061c102c746310d"} Oct 30 13:30:13.718468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.718447 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7"] Oct 30 13:30:13.727661 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:13.727633 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b4bc3e_9381_47bc_b53a_608d6bbd15f4.slice/crio-6ac2fc4d2eb06c92ecca58a243dde1b7d2e1925d64cb5e7b693f974c640e290c WatchSource:0}: Error finding container 6ac2fc4d2eb06c92ecca58a243dde1b7d2e1925d64cb5e7b693f974c640e290c: Status 404 returned error can't find the container with id 6ac2fc4d2eb06c92ecca58a243dde1b7d2e1925d64cb5e7b693f974c640e290c Oct 30 13:30:13.729174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.729148 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f7ee8a-openshift-default-dockercfg-sjnlf\"" Oct 30 13:30:13.745017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.734090 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn"] Oct 30 13:30:13.745017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.738631 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq"] Oct 30 13:30:13.796054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796027 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxx8\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.796176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.796176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796153 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.796259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796188 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.796496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796468 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.796647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796537 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.796647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796639 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.797097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796818 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.797097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.796899 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.907497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.867980 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podStartSLOduration=1.867958313 podStartE2EDuration="1.867958313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:13.866091415 +0000 UTC m=+1589.732709602" watchObservedRunningTime="2026-10-30 13:30:13.867958313 +0000 UTC m=+1589.734576486" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909037 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909110 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909140 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909209 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909340 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909395 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909508 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxx8\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909584 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.909625 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.910136 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.910496 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.911144 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.913224 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.913841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.913796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.917628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.917572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.921915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.921874 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.951115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.950989 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.955823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.955694 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxx8\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8\") pod \"e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:13.984914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.984885 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:13.989184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:13.989160 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:14.028403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.028368 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:14.254386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.254357 5693 scope.go:122] "RemoveContainer" containerID="58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f" Oct 30 13:30:14.372793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.372759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:14.471186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.470975 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq"] Oct 30 13:30:14.473746 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:14.473719 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04aea9d_1efd_4626_9bea_8449107bf8b0.slice/crio-9b043ae455911d73a3c99d0e9f90a8705ec5fbf0dc1baa8173674007601c676e WatchSource:0}: Error finding container 9b043ae455911d73a3c99d0e9f90a8705ec5fbf0dc1baa8173674007601c676e: Status 404 returned error can't find the container with id 9b043ae455911d73a3c99d0e9f90a8705ec5fbf0dc1baa8173674007601c676e Oct 30 13:30:14.476466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.476433 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:14.476544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.476494 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:14.476544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.476524 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:14.658377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.658353 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 13:30:14.725459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.725156 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" event={"ID":"e04aea9d-1efd-4626-9bea-8449107bf8b0","Type":"ContainerStarted","Data":"6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a"} Oct 30 13:30:14.725459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.725218 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" event={"ID":"e04aea9d-1efd-4626-9bea-8449107bf8b0","Type":"ContainerStarted","Data":"9b043ae455911d73a3c99d0e9f90a8705ec5fbf0dc1baa8173674007601c676e"} Oct 30 13:30:14.727842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.727741 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" event={"ID":"0a45b9f7-0e77-4385-a4f5-a0193805c4fb","Type":"ContainerStarted","Data":"b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626"} Oct 30 13:30:14.727842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.727809 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" event={"ID":"0a45b9f7-0e77-4385-a4f5-a0193805c4fb","Type":"ContainerStarted","Data":"0ee387512ad453b5faa1102af7b345eee05511aff71bc1a500b22e63ab66a5b2"} Oct 30 13:30:14.728101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.727872 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:14.730785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.730499 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:14.731087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.731050 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" event={"ID":"31b4bc3e-9381-47bc-b53a-608d6bbd15f4","Type":"ContainerStarted","Data":"08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa"} Oct 30 13:30:14.731215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.731107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" event={"ID":"31b4bc3e-9381-47bc-b53a-608d6bbd15f4","Type":"ContainerStarted","Data":"6ac2fc4d2eb06c92ecca58a243dde1b7d2e1925d64cb5e7b693f974c640e290c"} Oct 30 13:30:14.732425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.731831 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:14.737023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.737000 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerStarted","Data":"31efb5e2d791bf9b06c56070a9640d8c300543c30bc73582b95ffc298263b6c1"} Oct 30 13:30:14.795216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.795152 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podStartSLOduration=1.795132248 podStartE2EDuration="1.795132248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:14.794162096 +0000 UTC m=+1590.660780266" watchObservedRunningTime="2026-10-30 13:30:14.795132248 +0000 UTC m=+1590.661750638" Oct 30 13:30:14.846244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.844610 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" podStartSLOduration=14.84457755 podStartE2EDuration="14.84457755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:14.841098291 +0000 UTC m=+1590.707716500" watchObservedRunningTime="2026-10-30 13:30:14.84457755 +0000 UTC m=+1590.711195726" Oct 30 13:30:14.891334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.891246 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" podStartSLOduration=2.891228378 podStartE2EDuration="2.891228378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:14.888389411 +0000 UTC m=+1590.755007584" watchObservedRunningTime="2026-10-30 13:30:14.891228378 +0000 UTC m=+1590.757846562" Oct 30 13:30:14.936091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.936030 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" podStartSLOduration=2.9360143069999998 podStartE2EDuration="2.936014307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:14.932203308 +0000 UTC m=+1590.798821509" watchObservedRunningTime="2026-10-30 13:30:14.936014307 +0000 UTC m=+1590.802632477" Oct 30 13:30:14.944676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.944642 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:14.991545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:14.991470 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:15.029543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.029508 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:15.036250 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.036220 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:15.253301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.253217 5693 scope.go:122] "RemoveContainer" containerID="ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f" Oct 30 13:30:15.254465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.254441 5693 scope.go:122] "RemoveContainer" containerID="ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f" Oct 30 13:30:15.748553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.748520 5693 generic.go:356] "Generic (PLEG): container finished" podID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerID="31efb5e2d791bf9b06c56070a9640d8c300543c30bc73582b95ffc298263b6c1" exitCode=60 Oct 30 13:30:15.749115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.748558 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerDied","Data":"31efb5e2d791bf9b06c56070a9640d8c300543c30bc73582b95ffc298263b6c1"} Oct 30 13:30:15.749238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.749151 5693 scope.go:122] "RemoveContainer" containerID="58c85187c9e1a8dfc21ec8cc2523d236bd412b57546310b2ed18c2794169dd8f" Oct 30 13:30:15.749574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.749555 5693 scope.go:122] "RemoveContainer" containerID="31efb5e2d791bf9b06c56070a9640d8c300543c30bc73582b95ffc298263b6c1" Oct 30 13:30:15.750256 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:15.749972 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd_odh-ai-gateway-infra(a9b0d9f1-27cc-493a-ba06-070052833d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" Oct 30 13:30:15.750256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.750083 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:15.750256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.750130 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:15.750789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.750716 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:30:15.751401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.751373 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:30:15.875786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.875752 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd"] Oct 30 13:30:15.995153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:15.994933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:16.256345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.256269 5693 scope.go:122] "RemoveContainer" containerID="8e77f8ca3ee27015b0618541a232cf6a9c6a8d735b0283a0b461b777b2189d49" Oct 30 13:30:16.759399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.759356 5693 generic.go:356] "Generic (PLEG): container finished" podID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerID="95c52538ab797ed6a79ddddb88e58e3ee7c427b2005fcad656f4dd972ee7bab7" exitCode=60 Oct 30 13:30:16.759794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.759439 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" event={"ID":"cd4c0756-15f8-4eeb-8fb5-079a2f903d72","Type":"ContainerDied","Data":"95c52538ab797ed6a79ddddb88e58e3ee7c427b2005fcad656f4dd972ee7bab7"} Oct 30 13:30:16.759794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.759516 5693 scope.go:122] "RemoveContainer" containerID="ed2141389ad423632705ed7284b09e9071ae5fd9ae85f3c4fde78dfcc8cfc14f" Oct 30 13:30:16.759976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.759959 5693 scope.go:122] "RemoveContainer" containerID="95c52538ab797ed6a79ddddb88e58e3ee7c427b2005fcad656f4dd972ee7bab7" Oct 30 13:30:16.760282 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:16.760265 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx_odh-ai-gateway-infra(cd4c0756-15f8-4eeb-8fb5-079a2f903d72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" Oct 30 13:30:16.762124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.762102 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" event={"ID":"8638cd7c-6d5f-422d-a039-b702647577ee","Type":"ContainerStarted","Data":"55c34255801794aefc7315237ee4750227bf4eaa7f2c1fd13a463167b75f6677"} Oct 30 13:30:16.764158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.764136 5693 generic.go:356] "Generic (PLEG): container finished" podID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerID="fd6e425fed2af145c14778407cd7c15038ed1943d815fc683a0c85f1258397cb" exitCode=60 Oct 30 13:30:16.764266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.764215 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" event={"ID":"4409f36d-9c89-40a6-a21e-78b57d90948f","Type":"ContainerDied","Data":"fd6e425fed2af145c14778407cd7c15038ed1943d815fc683a0c85f1258397cb"} Oct 30 13:30:16.764641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.764623 5693 scope.go:122] "RemoveContainer" containerID="fd6e425fed2af145c14778407cd7c15038ed1943d815fc683a0c85f1258397cb" Oct 30 13:30:16.764958 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:16.764938 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6_odh-ai-gateway-infra(4409f36d-9c89-40a6-a21e-78b57d90948f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" Oct 30 13:30:16.772872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.772851 5693 scope.go:122] "RemoveContainer" containerID="ef909c93fa175ed8325cb803718bd61f85e026768fbba5490790996edc35953f" Oct 30 13:30:16.774930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.774911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:16.948776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.948747 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:16.977804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:16.977751 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" podStartSLOduration=16.977737241 podStartE2EDuration="16.977737241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:16.97426091 +0000 UTC m=+1592.840879083" watchObservedRunningTime="2026-10-30 13:30:16.977737241 +0000 UTC m=+1592.844355413" Oct 30 13:30:17.001486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.001459 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:17.028484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.028448 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp\" (UniqueName: \"kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp\") pod \"a9b0d9f1-27cc-493a-ba06-070052833d54\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " Oct 30 13:30:17.028680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.028500 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca\") pod \"a9b0d9f1-27cc-493a-ba06-070052833d54\" (UID: \"a9b0d9f1-27cc-493a-ba06-070052833d54\") " Oct 30 13:30:17.029725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.029695 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca" pod "a9b0d9f1-27cc-493a-ba06-070052833d54" (UID: "a9b0d9f1-27cc-493a-ba06-070052833d54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.031473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.031440 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp" pod "a9b0d9f1-27cc-493a-ba06-070052833d54" (UID: "a9b0d9f1-27cc-493a-ba06-070052833d54"). InnerVolumeSpecName "kube-api-access-4p9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.130609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.130505 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p9jp\" (UniqueName: \"kubernetes.io/projected/a9b0d9f1-27cc-493a-ba06-070052833d54-kube-api-access-4p9jp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.130609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.130546 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b0d9f1-27cc-493a-ba06-070052833d54-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.641956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.641898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:17.775682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.775650 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" Oct 30 13:30:17.776307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.775642 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd" event={"ID":"a9b0d9f1-27cc-493a-ba06-070052833d54","Type":"ContainerDied","Data":"0decc089f493bee3113663d5da5cb70b543adf60ad3bc6d740833c1bfc63554b"} Oct 30 13:30:17.776307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.775812 5693 scope.go:122] "RemoveContainer" containerID="31efb5e2d791bf9b06c56070a9640d8c300543c30bc73582b95ffc298263b6c1" Oct 30 13:30:17.782131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.781768 5693 generic.go:356] "Generic (PLEG): container finished" podID="8638cd7c-6d5f-422d-a039-b702647577ee" containerID="55c34255801794aefc7315237ee4750227bf4eaa7f2c1fd13a463167b75f6677" exitCode=60 Oct 30 13:30:17.782131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.781923 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" event={"ID":"8638cd7c-6d5f-422d-a039-b702647577ee","Type":"ContainerDied","Data":"55c34255801794aefc7315237ee4750227bf4eaa7f2c1fd13a463167b75f6677"} Oct 30 13:30:17.782841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.782817 5693 scope.go:122] "RemoveContainer" containerID="55c34255801794aefc7315237ee4750227bf4eaa7f2c1fd13a463167b75f6677" Oct 30 13:30:17.783506 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:17.783478 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4_odh-ai-gateway-infra(8638cd7c-6d5f-422d-a039-b702647577ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" Oct 30 13:30:17.795078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.795052 5693 scope.go:122] "RemoveContainer" containerID="8e77f8ca3ee27015b0618541a232cf6a9c6a8d735b0283a0b461b777b2189d49" Oct 30 13:30:17.834225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.834197 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx"] Oct 30 13:30:17.904629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.897890 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6"] Oct 30 13:30:17.937826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.937788 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd"] Oct 30 13:30:17.947575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.947539 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:17.976862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:17.976825 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889450-s5gpd"] Oct 30 13:30:18.004072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.004042 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4"] Oct 30 13:30:18.089632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.087667 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:18.093734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.093706 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:18.251993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.251964 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca\") pod \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " Oct 30 13:30:18.252209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252021 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4\" (UniqueName: \"kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4\") pod \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\" (UID: \"cd4c0756-15f8-4eeb-8fb5-079a2f903d72\") " Oct 30 13:30:18.252209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252123 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t\" (UniqueName: \"kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t\") pod \"4409f36d-9c89-40a6-a21e-78b57d90948f\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " Oct 30 13:30:18.252209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252173 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca\") pod \"4409f36d-9c89-40a6-a21e-78b57d90948f\" (UID: \"4409f36d-9c89-40a6-a21e-78b57d90948f\") " Oct 30 13:30:18.252569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252415 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca" pod "cd4c0756-15f8-4eeb-8fb5-079a2f903d72" (UID: "cd4c0756-15f8-4eeb-8fb5-079a2f903d72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.252685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252582 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca" pod "4409f36d-9c89-40a6-a21e-78b57d90948f" (UID: "4409f36d-9c89-40a6-a21e-78b57d90948f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.252939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252917 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.253001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.252947 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4409f36d-9c89-40a6-a21e-78b57d90948f-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.254361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.254337 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t" pod "4409f36d-9c89-40a6-a21e-78b57d90948f" (UID: "4409f36d-9c89-40a6-a21e-78b57d90948f"). InnerVolumeSpecName "kube-api-access-7ft8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.254500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.254370 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4" pod "cd4c0756-15f8-4eeb-8fb5-079a2f903d72" (UID: "cd4c0756-15f8-4eeb-8fb5-079a2f903d72"). InnerVolumeSpecName "kube-api-access-hz4d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.257306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.257283 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" path="/var/lib/kubelet/pods/a9b0d9f1-27cc-493a-ba06-070052833d54/volumes" Oct 30 13:30:18.354893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.354854 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4d4\" (UniqueName: \"kubernetes.io/projected/cd4c0756-15f8-4eeb-8fb5-079a2f903d72-kube-api-access-hz4d4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.354893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.354892 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ft8t\" (UniqueName: \"kubernetes.io/projected/4409f36d-9c89-40a6-a21e-78b57d90948f-kube-api-access-7ft8t\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.483431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.483382 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:18.632686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.632578 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz"] Oct 30 13:30:18.633281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633265 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:18.633321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633282 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:18.633321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633294 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633301 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633323 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633329 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633337 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633341 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633368 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633373 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633381 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633386 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633393 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633397 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633483 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633491 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633501 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633507 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633514 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633520 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633528 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:18.633871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.633535 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:18.638628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.638608 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.654775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.654748 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:18.656767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.656746 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f45f04-dockercfg-26pn6\"" Oct 30 13:30:18.661704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.661489 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz"] Oct 30 13:30:18.663252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.663235 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f45f04\"" Oct 30 13:30:18.790671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.772549 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.812073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.812035 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z22h\" (UniqueName: \"kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h\") pod \"payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.985014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.958774 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.985014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.966180 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z22h\" (UniqueName: \"kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h\") pod \"payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.985014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.976095 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.992945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.991058 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z22h\" (UniqueName: \"kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h\") pod \"payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:18.996251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.996219 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" Oct 30 13:30:18.997095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.997025 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6" event={"ID":"4409f36d-9c89-40a6-a21e-78b57d90948f","Type":"ContainerDied","Data":"13f6ef545c181fbb6d4cb6a8d897c065da180329901480cb74f8f0b98e562c2f"} Oct 30 13:30:18.997095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:18.997096 5693 scope.go:122] "RemoveContainer" containerID="fd6e425fed2af145c14778407cd7c15038ed1943d815fc683a0c85f1258397cb" Oct 30 13:30:19.000848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.000817 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" event={"ID":"cd4c0756-15f8-4eeb-8fb5-079a2f903d72","Type":"ContainerDied","Data":"df8fe2349251bd6cdd220c23cdd11ece718ba12b938c41b6e983a4f10df4d6db"} Oct 30 13:30:19.000975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.000927 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx" Oct 30 13:30:19.020912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.020884 5693 scope.go:122] "RemoveContainer" containerID="95c52538ab797ed6a79ddddb88e58e3ee7c427b2005fcad656f4dd972ee7bab7" Oct 30 13:30:19.064797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.064768 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx"] Oct 30 13:30:19.076948 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.074778 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889450-slqxx"] Oct 30 13:30:19.297827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.288054 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:19.303410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.303375 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6"] Oct 30 13:30:19.329004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.328971 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889450-dhkx6"] Oct 30 13:30:19.379858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.379826 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:19.502742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.502711 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca\") pod \"8638cd7c-6d5f-422d-a039-b702647577ee\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " Oct 30 13:30:19.502943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.502824 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt\" (UniqueName: \"kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt\") pod \"8638cd7c-6d5f-422d-a039-b702647577ee\" (UID: \"8638cd7c-6d5f-422d-a039-b702647577ee\") " Oct 30 13:30:19.503127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.503100 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca" pod "8638cd7c-6d5f-422d-a039-b702647577ee" (UID: "8638cd7c-6d5f-422d-a039-b702647577ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:19.503478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.503459 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8638cd7c-6d5f-422d-a039-b702647577ee-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.504987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.504959 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt" pod "8638cd7c-6d5f-422d-a039-b702647577ee" (UID: "8638cd7c-6d5f-422d-a039-b702647577ee"). InnerVolumeSpecName "kube-api-access-9kmgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:19.614651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.614556 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmgt\" (UniqueName: \"kubernetes.io/projected/8638cd7c-6d5f-422d-a039-b702647577ee-kube-api-access-9kmgt\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.663815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:19.663786 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz"] Oct 30 13:30:19.665612 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:19.665573 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33baf68d_9646_4635_92d6_7bc8cc11fba2.slice/crio-cc44ced6bc345b36ecc13edef9797e0ba161028009cfcea7cff80267485a10b8 WatchSource:0}: Error finding container cc44ced6bc345b36ecc13edef9797e0ba161028009cfcea7cff80267485a10b8: Status 404 returned error can't find the container with id cc44ced6bc345b36ecc13edef9797e0ba161028009cfcea7cff80267485a10b8 Oct 30 13:30:20.025619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.024752 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" event={"ID":"33baf68d-9646-4635-92d6-7bc8cc11fba2","Type":"ContainerStarted","Data":"574e66132e5a5c77ccf81ed7e93d5dfdcdcaa1bc3cd21a13e1488c350adbb69b"} Oct 30 13:30:20.025619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.024817 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" event={"ID":"33baf68d-9646-4635-92d6-7bc8cc11fba2","Type":"ContainerStarted","Data":"cc44ced6bc345b36ecc13edef9797e0ba161028009cfcea7cff80267485a10b8"} Oct 30 13:30:20.025619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.024847 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:20.029645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.027648 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" event={"ID":"8638cd7c-6d5f-422d-a039-b702647577ee","Type":"ContainerDied","Data":"a51596d03cf01f8ff48ef892f109388b56956ad09b9dd8f150e19ec47b81d2b7"} Oct 30 13:30:20.029645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.027718 5693 scope.go:122] "RemoveContainer" containerID="55c34255801794aefc7315237ee4750227bf4eaa7f2c1fd13a463167b75f6677" Oct 30 13:30:20.029645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.027885 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4" Oct 30 13:30:20.110354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.110293 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" podStartSLOduration=2.110274233 podStartE2EDuration="2.110274233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:20.103529098 +0000 UTC m=+1595.970147284" watchObservedRunningTime="2026-10-30 13:30:20.110274233 +0000 UTC m=+1595.976892407" Oct 30 13:30:20.160662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.160627 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4"] Oct 30 13:30:20.170692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.170656 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889450-6grt4"] Oct 30 13:30:20.371695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.369201 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" path="/var/lib/kubelet/pods/4409f36d-9c89-40a6-a21e-78b57d90948f/volumes" Oct 30 13:30:20.372812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.372785 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" path="/var/lib/kubelet/pods/8638cd7c-6d5f-422d-a039-b702647577ee/volumes" Oct 30 13:30:20.381072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.373673 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4c0756-15f8-4eeb-8fb5-079a2f903d72" path="/var/lib/kubelet/pods/cd4c0756-15f8-4eeb-8fb5-079a2f903d72/volumes" Oct 30 13:30:20.771854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.771816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:20.851663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.851618 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9"] Oct 30 13:30:20.852705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852680 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.852705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852706 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.852926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852750 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:20.852926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852759 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:20.852926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852781 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:20.852926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852789 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b0d9f1-27cc-493a-ba06-070052833d54" containerName="cleanup" Oct 30 13:30:20.852926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852818 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.852926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.852847 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.853271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.853013 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.853271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.853026 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4409f36d-9c89-40a6-a21e-78b57d90948f" containerName="cleanup" Oct 30 13:30:20.853271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.853044 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.853271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.853058 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:20.871459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.866121 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:20.895616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.894101 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f7ee8a\"" Oct 30 13:30:20.895616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.894233 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f7ee8a-dockercfg-8nr4j\"" Oct 30 13:30:20.901848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.901822 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9"] Oct 30 13:30:20.906890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.906857 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qt8\" (UniqueName: \"kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8\") pod \"payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:20.907104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:20.907085 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:21.027694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.003018 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:21.054054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.031171 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:21.054054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.031422 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qt8\" (UniqueName: \"kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8\") pod \"payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:21.054054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.032766 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:21.274901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.271376 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qt8\" (UniqueName: \"kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8\") pod \"payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:21.516643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.515799 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:21.704849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:21.702486 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:22.114281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:22.114067 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9"] Oct 30 13:30:22.122901 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:22.122870 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7772c4_fda4_4bf7_9cd1_a419d07aff95.slice/crio-f6c86e3d805b9823732ba6c7a0af8d653451b68896728dd9879520a9caafc5fe WatchSource:0}: Error finding container f6c86e3d805b9823732ba6c7a0af8d653451b68896728dd9879520a9caafc5fe: Status 404 returned error can't find the container with id f6c86e3d805b9823732ba6c7a0af8d653451b68896728dd9879520a9caafc5fe Oct 30 13:30:22.276033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:22.276002 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:22.276199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:22.276049 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" event={"ID":"0b7772c4-fda4-4bf7-9cd1-a419d07aff95","Type":"ContainerStarted","Data":"cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d"} Oct 30 13:30:22.276199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:22.276070 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" event={"ID":"0b7772c4-fda4-4bf7-9cd1-a419d07aff95","Type":"ContainerStarted","Data":"f6c86e3d805b9823732ba6c7a0af8d653451b68896728dd9879520a9caafc5fe"} Oct 30 13:30:22.317368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:22.317292 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" podStartSLOduration=2.317262188 podStartE2EDuration="2.317262188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:22.314388744 +0000 UTC m=+1598.181006918" watchObservedRunningTime="2026-10-30 13:30:22.317262188 +0000 UTC m=+1598.183880361" Oct 30 13:30:23.077275 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:23.077236 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:30:23.895623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:23.885805 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:25.761641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:25.756308 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:25.783636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:25.769866 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 13:30:26.770112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:26.770082 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 13:30:27.642915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:27.642830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:29.443145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:29.443108 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:31.197175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.194699 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:31.284980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.284948 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:30:31.358346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.358312 5693 generic.go:356] "Generic (PLEG): container finished" podID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerID="30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4" exitCode=0 Oct 30 13:30:31.358512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.358383 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" event={"ID":"cf039ca7-5fc7-4310-8c29-685bc70445d1","Type":"ContainerDied","Data":"30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4"} Oct 30 13:30:31.358960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.358936 5693 scope.go:122] "RemoveContainer" containerID="30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4" Oct 30 13:30:31.361020 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.360980 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:31.361112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.361077 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:31.361160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:31.361121 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:32.367350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:32.367315 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" event={"ID":"cf039ca7-5fc7-4310-8c29-685bc70445d1","Type":"ContainerStarted","Data":"8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793"} Oct 30 13:30:32.777758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:32.777716 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:30:32.778559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:32.778533 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:30:32.778736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:32.778585 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:30:32.854146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:32.854102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:33.001650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.001588 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7"] Oct 30 13:30:33.002780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.002758 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:33.002837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.002784 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8638cd7c-6d5f-422d-a039-b702647577ee" containerName="cleanup" Oct 30 13:30:33.007325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.007304 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.056478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.056397 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-552cc3-serving-cert\"" Oct 30 13:30:33.058717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.058685 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:33.068625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.064994 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7"] Oct 30 13:30:33.173373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.173337 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.173552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.173405 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.173552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.173443 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6cs\" (UniqueName: \"kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.173552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.173470 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.217044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.217016 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:30:33.257623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.253332 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:33.296814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.291058 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.296814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.291163 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.296814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.291196 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6cs\" (UniqueName: \"kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.296814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.291228 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.296814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.293324 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.310741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.310247 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.320874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.319158 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.320874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.320029 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 13:30:33.355618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.351641 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6cs\" (UniqueName: \"kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs\") pod \"maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.476920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.476889 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:33.622146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.622064 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:33.754652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.751992 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:30:33.754652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.752052 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:30:33.983942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:33.983886 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7"] Oct 30 13:30:34.004953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:34.004925 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:30:34.423508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:34.423418 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" event={"ID":"5dedcc23-b119-473d-8c63-cfe2ec549680","Type":"ContainerStarted","Data":"323bf6f671567efceade0f19fdb9d0a8b7963b5f2d0105c2253e3d892f2a39f0"} Oct 30 13:30:34.749624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:34.749025 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:30:34.749624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:34.749091 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:30:35.119692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.119104 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:35.273269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.273235 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:35.494336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.494298 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:35.496343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.496308 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" event={"ID":"5dedcc23-b119-473d-8c63-cfe2ec549680","Type":"ContainerStarted","Data":"7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6"} Oct 30 13:30:35.497916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.497860 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:35.751692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.746206 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:30:35.751692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:35.746285 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:30:36.747142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:36.747103 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:30:36.747450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:36.747420 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:30:37.152687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:37.151839 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:37.748372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:37.748327 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:30:37.749707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:37.749682 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 13:30:37.819780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:37.819720 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" podStartSLOduration=5.34373912 podStartE2EDuration="5.819705211s" totalImagesPullingTime="475.966091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:35.55057471 +0000 UTC m=+1611.417192884" watchObservedRunningTime="2026-10-30 13:30:37.819705211 +0000 UTC m=+1613.686323385" Oct 30 13:30:38.193285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:38.193195 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:38.875239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:38.875203 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:40.494360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.494325 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl"] Oct 30 13:30:40.499070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.499045 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.529899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.529871 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f45f04-serving-cert\"" Oct 30 13:30:40.541850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.541818 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9kv\" (UniqueName: \"kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.542014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.541881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.542014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.541993 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.542143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.542029 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.543475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.543450 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl"] Oct 30 13:30:40.643198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.643161 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9kv\" (UniqueName: \"kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.643412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.643214 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.643412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.643249 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.643412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.643267 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.644095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.644070 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.645848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.645822 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.645962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.645866 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.664514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.664490 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9kv\" (UniqueName: \"kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv\") pod \"maas-api-e2e-autores-f45f04-9458f684f-2k8rl\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.733624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.730983 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:40.816325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.816236 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:40.915825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.915790 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4"] Oct 30 13:30:40.923168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.923138 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.940816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.940785 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz"] Oct 30 13:30:40.946383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.946360 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:40.947333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.947313 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52af49-openshift-default-dockercfg-tbzp6\"" Oct 30 13:30:40.949139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949113 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.949264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949248 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.949371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949352 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdzb\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.949520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949426 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.949685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949516 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.949685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949576 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.949685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949667 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.950004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949713 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.950004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.949837 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:40.956615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.956573 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4"] Oct 30 13:30:40.959206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.958473 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:40.982252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:40.982221 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz"] Oct 30 13:30:41.051219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051184 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051239 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051291 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051312 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051446 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051535 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051641 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051677 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051714 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdzb\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051836 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051863 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051929 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.051999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051964 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.051999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.051990 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.052230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.052230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052175 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.052316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052254 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.052539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.052648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052628 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.052712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052691 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.052749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.052729 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.054040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.054018 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.054822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.054803 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.081082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.081001 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.081946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.081915 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdzb\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.135372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.135349 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl"] Oct 30 13:30:41.138049 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:41.138022 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeab246f_c9b1_4346_acbe_49021f319edb.slice/crio-7b9b2e438f8b25d0ebf4e90119477b2e92a8641c5d95737f8c98125d244bd82f WatchSource:0}: Error finding container 7b9b2e438f8b25d0ebf4e90119477b2e92a8641c5d95737f8c98125d244bd82f: Status 404 returned error can't find the container with id 7b9b2e438f8b25d0ebf4e90119477b2e92a8641c5d95737f8c98125d244bd82f Oct 30 13:30:41.154080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154040 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154124 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154161 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154214 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154287 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154314 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154387 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154441 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154490 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154566 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154729 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154821 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.154897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.154876 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.155112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.155088 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.156725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.156706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.157050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.157032 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.176653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.176593 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.176886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.176870 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.249745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.249716 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:41.267353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.267329 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:41.547234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.547202 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" event={"ID":"beab246f-c9b1-4346-acbe-49021f319edb","Type":"ContainerStarted","Data":"7b9b2e438f8b25d0ebf4e90119477b2e92a8641c5d95737f8c98125d244bd82f"} Oct 30 13:30:41.592389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.592362 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4"] Oct 30 13:30:41.594503 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:41.594476 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ffc4c1_3d5f_4be5_a663_e5aa035fce83.slice/crio-3d7f5e5e780f5f475ab2b54c0de21df05ea22950859c857df92b9cc7bd55ad19 WatchSource:0}: Error finding container 3d7f5e5e780f5f475ab2b54c0de21df05ea22950859c857df92b9cc7bd55ad19: Status 404 returned error can't find the container with id 3d7f5e5e780f5f475ab2b54c0de21df05ea22950859c857df92b9cc7bd55ad19 Oct 30 13:30:41.597268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.597236 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:41.597353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.597302 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:41.597353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.597330 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:41.607916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.607893 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz"] Oct 30 13:30:41.612330 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:30:41.612308 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d94421b_6340_457d_a007_3324fc757d63.slice/crio-25da15dfd3b6520999f7bf60dd82dd2f7449fc5bf9ddab61b5c92ada314a52fd WatchSource:0}: Error finding container 25da15dfd3b6520999f7bf60dd82dd2f7449fc5bf9ddab61b5c92ada314a52fd: Status 404 returned error can't find the container with id 25da15dfd3b6520999f7bf60dd82dd2f7449fc5bf9ddab61b5c92ada314a52fd Oct 30 13:30:41.614980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.614947 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:41.615084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.615025 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:41.615084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:41.615070 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:42.520429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.520394 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 13:30:42.559545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.559502 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" event={"ID":"01ffc4c1-3d5f-4be5-a663-e5aa035fce83","Type":"ContainerStarted","Data":"cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a"} Oct 30 13:30:42.560112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.559568 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" event={"ID":"01ffc4c1-3d5f-4be5-a663-e5aa035fce83","Type":"ContainerStarted","Data":"3d7f5e5e780f5f475ab2b54c0de21df05ea22950859c857df92b9cc7bd55ad19"} Oct 30 13:30:42.569953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.569902 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" event={"ID":"0d94421b-6340-457d-a007-3324fc757d63","Type":"ContainerStarted","Data":"37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71"} Oct 30 13:30:42.570229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.570202 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" event={"ID":"0d94421b-6340-457d-a007-3324fc757d63","Type":"ContainerStarted","Data":"25da15dfd3b6520999f7bf60dd82dd2f7449fc5bf9ddab61b5c92ada314a52fd"} Oct 30 13:30:42.573591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.573558 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" event={"ID":"beab246f-c9b1-4346-acbe-49021f319edb","Type":"ContainerStarted","Data":"bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60"} Oct 30 13:30:42.574469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.574447 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:42.609770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.609707 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podStartSLOduration=2.609687261 podStartE2EDuration="2.609687261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:42.60456803 +0000 UTC m=+1618.471186202" watchObservedRunningTime="2026-10-30 13:30:42.609687261 +0000 UTC m=+1618.476305434" Oct 30 13:30:42.731624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.712544 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:42.756636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.751986 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" podStartSLOduration=2.442018696 podStartE2EDuration="2.751961968s" totalImagesPullingTime="309.943272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:42.751207646 +0000 UTC m=+1618.617825820" watchObservedRunningTime="2026-10-30 13:30:42.751961968 +0000 UTC m=+1618.618580169" Oct 30 13:30:42.756636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:42.753539 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podStartSLOduration=2.75352562 podStartE2EDuration="2.75352562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:42.683973987 +0000 UTC m=+1618.550592161" watchObservedRunningTime="2026-10-30 13:30:42.75352562 +0000 UTC m=+1618.620143794" Oct 30 13:30:43.252623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:43.252036 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:43.252623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:43.252209 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:43.252623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:43.252259 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:43.276654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:43.276419 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:43.276654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:43.276572 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:43.276942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:43.276652 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:44.279626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:44.268947 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:44.279626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:44.269020 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:44.281962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:44.281931 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:44.282174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:44.282144 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:44.765620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:44.763136 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:44.948628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:44.947913 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:45.272619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.272205 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:45.272619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.272269 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:45.288623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.274110 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:45.288623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.274156 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:45.870854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.865589 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm"] Oct 30 13:30:45.926633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.916703 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:45.978853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:45.968315 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.033667 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.033719 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.033875 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.033954 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.034024 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.034051 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.034102 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.038622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.034131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5tv\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.063624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.047004 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm"] Oct 30 13:30:46.208637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.138909 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.450699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.268818 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:46.450699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.441764 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:46.450699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.285106 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:46.450699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.444207 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:46.477481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.477446 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.477760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.477738 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.478044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.478013 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.479210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.478061 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.481848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.481812 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.482066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.478782 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.482637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.482380 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.482799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.482782 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.490104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.483407 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.490345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.490311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5tv\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.494884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.494855 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.574622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.520179 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.574622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.537684 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.574622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.550262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.574622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.554873 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.606493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.605673 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5tv\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.618287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.618257 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:46.759912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:46.739324 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:47.251205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.250318 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:47.251205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.250370 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:47.293312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.284347 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:47.293312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.284409 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:47.603679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.603049 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm"] Oct 30 13:30:47.640693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.632859 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:47.668623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.668362 5693 scope.go:122] "RemoveContainer" containerID="9f0d325304a2cc2c716b0a666f9613493f3ad14e90f6db9fe5ce745e122294fd" Oct 30 13:30:47.708014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.696122 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:47.708014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.696222 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:47.708014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:47.696269 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:48.013131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:48.012472 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" event={"ID":"188c84ed-c1ed-4846-8f5f-feb3790fb8ad","Type":"ContainerStarted","Data":"faadf1b8dbe8ed727b57e0290207490b63a0920c11d2dbe5adf73368fe8bae8b"} Oct 30 13:30:48.275795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:48.270067 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:48.275795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:48.270132 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:48.275795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:48.271860 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:48.275795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:48.271906 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:49.023089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.023050 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" event={"ID":"188c84ed-c1ed-4846-8f5f-feb3790fb8ad","Type":"ContainerStarted","Data":"2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca"} Oct 30 13:30:49.110370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.110308 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podStartSLOduration=4.110290727 podStartE2EDuration="4.110290727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:49.097915215 +0000 UTC m=+1624.964533388" watchObservedRunningTime="2026-10-30 13:30:49.110290727 +0000 UTC m=+1624.976908901" Oct 30 13:30:49.262628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.258958 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:30:49.262628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.259020 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:30:49.271674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.268347 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 13:30:49.271674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.268405 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 13:30:49.650641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.626136 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:49.704622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.702783 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:30:49.748922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.748886 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:49.765865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.761495 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:49.765865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:49.761554 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:50.742010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:50.741968 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:50.742615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:50.742571 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:51.169734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.166930 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:51.169734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.168531 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:51.182230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.182190 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:30:51.284621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.280723 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:30:51.284621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.280787 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:30:51.307443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.304175 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:51.307443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.305352 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:51.308722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.308700 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:30:51.742635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.729563 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:51.768668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.741034 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:51.768928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:51.768895 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:52.759888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:52.757821 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:52.759888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:52.757895 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:53.124641 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:30:53.124519 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:30:53.727684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:53.724480 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:53.743561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:53.743002 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:53.743561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:53.743067 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:54.447620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:54.440058 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:54.747622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:54.740990 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:54.747622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:54.741067 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:55.418558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:55.418525 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:55.695801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:55.695717 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:30:55.696223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:55.695821 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:30:55.741719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:55.741683 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 13:30:55.741899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:55.741736 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 13:30:56.745119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:56.745087 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:56.745529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:56.745499 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:56.746167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:56.746147 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:30:57.144662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.144441 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:57.376175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.376140 5693 generic.go:356] "Generic (PLEG): container finished" podID="34a928a3-4554-425d-a2e5-38a118e6729e" containerID="0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591" exitCode=0 Oct 30 13:30:57.376390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.376211 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" event={"ID":"34a928a3-4554-425d-a2e5-38a118e6729e","Type":"ContainerDied","Data":"0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591"} Oct 30 13:30:57.376714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.376692 5693 scope.go:122] "RemoveContainer" containerID="0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591" Oct 30 13:30:57.378344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.378313 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:57.378425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.378388 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:57.378468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:57.378424 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:58.385675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:58.385635 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" event={"ID":"34a928a3-4554-425d-a2e5-38a118e6729e","Type":"ContainerStarted","Data":"3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d"} Oct 30 13:30:58.665377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:58.665289 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:30:58.665558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:58.665532 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 13:30:58.665646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:58.665588 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 13:30:58.966674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:58.966563 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:59.665721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:59.665689 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 13:30:59.666197 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:30:59.665755 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 13:31:00.666036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:00.665994 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 13:31:00.666455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:00.666060 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 13:31:00.792624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:00.784550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:01.666050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:01.666014 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 13:31:01.666466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:01.666081 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 13:31:02.665775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:02.665733 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 13:31:02.665962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:02.665812 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 13:31:02.773460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:02.773417 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:03.783359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:03.783318 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:31:03.784427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:03.784256 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:31:03.784977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:03.784957 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 13:31:04.764445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:04.764408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:05.058391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:05.058253 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:06.294976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:06.294865 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:06.665782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:06.665694 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:07.361644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:07.359225 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:08.414623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:08.404557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:10.504060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:10.504025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:11.508513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:11.508398 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:11.645397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:11.645251 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:12.568155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:12.568119 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:12.837302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:12.837102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:14.979626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:14.971098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:16.847273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:16.847238 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:18.657348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.657266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:18.802240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.802144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:18.832483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.832438 5693 generic.go:356] "Generic (PLEG): container finished" podID="d687829a-5d4b-469f-b1a3-262f612892a0" containerID="b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737" exitCode=0 Oct 30 13:31:18.832732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.832512 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" event={"ID":"d687829a-5d4b-469f-b1a3-262f612892a0","Type":"ContainerDied","Data":"b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737"} Oct 30 13:31:18.833114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.833095 5693 scope.go:122] "RemoveContainer" containerID="b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737" Oct 30 13:31:18.835026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.834993 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:18.835116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.835067 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:18.835116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:18.835100 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:19.113269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.113144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:19.627620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.619558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:19.789622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.786415 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:19.943290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.943180 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" event={"ID":"d687829a-5d4b-469f-b1a3-262f612892a0","Type":"ContainerStarted","Data":"3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89"} Oct 30 13:31:19.960621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.953397 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:31:19.960621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.953514 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:19.960621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:19.953557 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:20.687820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:20.687779 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:20.953771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:20.953237 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:20.953771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:20.953300 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:21.955804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:21.955636 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:21.955804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:21.955696 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:22.659639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.659530 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:22.953490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.953132 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:22.953490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.953210 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:22.977094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.977058 5693 generic.go:356] "Generic (PLEG): container finished" podID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerID="d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a" exitCode=0 Oct 30 13:31:22.977553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.977125 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" event={"ID":"61a4c160-ebaf-40a4-a5b5-f508d2aaba50","Type":"ContainerDied","Data":"d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a"} Oct 30 13:31:22.977669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.977652 5693 scope.go:122] "RemoveContainer" containerID="d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a" Oct 30 13:31:22.979183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.979158 5693 generic.go:356] "Generic (PLEG): container finished" podID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerID="6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a" exitCode=0 Oct 30 13:31:22.979270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.979220 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" event={"ID":"e04aea9d-1efd-4626-9bea-8449107bf8b0","Type":"ContainerDied","Data":"6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a"} Oct 30 13:31:22.979431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.979401 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:22.979487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.979474 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:22.979530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.979512 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:22.979582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.979569 5693 scope.go:122] "RemoveContainer" containerID="6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a" Oct 30 13:31:22.981186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.981157 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:22.981314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.981236 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:22.981314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:22.981294 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:23.089795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:23.089712 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:23.108127 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:31:23.108094 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:31:23.282655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:23.279343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:23.963621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:23.963024 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:23.963621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:23.963086 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:24.012527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.012491 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" event={"ID":"61a4c160-ebaf-40a4-a5b5-f508d2aaba50","Type":"ContainerStarted","Data":"5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85"} Oct 30 13:31:24.014453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.014427 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" event={"ID":"e04aea9d-1efd-4626-9bea-8449107bf8b0","Type":"ContainerStarted","Data":"ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465"} Oct 30 13:31:24.030618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.030103 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:31:24.030618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.030209 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:24.030618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.030251 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:24.769623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.759118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:25.010622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:24.990526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.190480 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.190542 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.191526 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.191563 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.199644 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.199823 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 13:31:25.229577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.199861 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 13:31:25.395361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.246929 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:25.663672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.662245 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:31:25.808627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.807767 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:31:25.846630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.846162 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:31:25.962677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.961841 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 13:31:25.962677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.961906 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 13:31:25.990750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.986726 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 13:31:25.990750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:25.986788 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 13:31:26.074759 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:26.074720 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:26.075047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:26.075022 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:26.991622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:26.988947 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:31:26.997627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:26.997583 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 13:31:26.997921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:26.997674 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 13:31:26.998135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:26.998105 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 13:31:27.032043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:27.031995 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:27.032364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:27.032332 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:27.124290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:27.123947 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:27.985260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:27.985214 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 13:31:27.985477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:27.985272 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 13:31:28.030634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.030500 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:28.030634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.030618 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:28.482679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.462466 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx"] Oct 30 13:31:28.494623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.493710 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx"] Oct 30 13:31:28.494623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.493826 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.524852 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.524829 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cab6c5\"/\"default-dockercfg-c4jbj\"" Oct 30 13:31:28.525020 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.524876 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cab6c5\"/\"auth-test-model-cab6c5-kserve-self-signed-certs\"" Oct 30 13:31:28.540792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.540760 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cab6c5\"/\"kube-root-ca.crt\"" Oct 30 13:31:28.541082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.541061 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cab6c5\"/\"openshift-service-ca.crt\"" Oct 30 13:31:28.610921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.610887 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.611094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.610934 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.611143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.611090 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.611143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.611131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggsmq\" (UniqueName: \"kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.611257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.611243 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.611360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.611347 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.645624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.645283 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:28.713202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713166 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.713484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713469 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.713643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713518 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.713780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713763 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.713873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713809 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.714520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713933 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.714520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.713971 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggsmq\" (UniqueName: \"kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.714520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.714230 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.714520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.714467 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.715724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.715575 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.717101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.717080 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.739821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.739706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggsmq\" (UniqueName: \"kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq\") pod \"auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.813168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.812878 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:28.996693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.991426 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 13:31:28.996693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:28.991485 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 13:31:29.050624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.048853 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:29.050624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.048914 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:29.379505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.379174 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx"] Oct 30 13:31:29.382992 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:31:29.382929 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7db209e0_95d6_4b63_848c_2a83c021c673.slice/crio-bab41a1f654f60daa77171fb40187e34679680bf0e1ed85961ceb280fa460026 WatchSource:0}: Error finding container bab41a1f654f60daa77171fb40187e34679680bf0e1ed85961ceb280fa460026: Status 404 returned error can't find the container with id bab41a1f654f60daa77171fb40187e34679680bf0e1ed85961ceb280fa460026 Oct 30 13:31:29.503626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.499252 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" event={"ID":"7db209e0-95d6-4b63-848c-2a83c021c673","Type":"ContainerStarted","Data":"bab41a1f654f60daa77171fb40187e34679680bf0e1ed85961ceb280fa460026"} Oct 30 13:31:29.503626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.503174 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:29.985177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.985143 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 13:31:29.985370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:29.985214 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 13:31:30.035624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.030948 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 13:31:30.035624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.031013 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 13:31:30.508648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.508567 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" event={"ID":"7db209e0-95d6-4b63-848c-2a83c021c673","Type":"ContainerStarted","Data":"67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6"} Oct 30 13:31:30.548921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.548847 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" podStartSLOduration=2.548825939 podStartE2EDuration="2.548825939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:30.547092719 +0000 UTC m=+1666.413710906" watchObservedRunningTime="2026-10-30 13:31:30.548825939 +0000 UTC m=+1666.415444126" Oct 30 13:31:30.749873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.749837 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:31:30.750063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.749930 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:31:30.816159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.816074 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:30.850345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:30.850312 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:31.067672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:31.064024 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:31:31.104142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:31.104110 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:31:31.442161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:31.441951 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:31.539444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:31.539417 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:31:31.540179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:31.540157 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 13:31:33.076003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:33.075958 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:34.176856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:34.176824 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:34.743585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:34.743554 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:36.427969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:36.427934 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:38.067622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:38.064192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:38.815686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:38.815649 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:38.828678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:38.828651 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:39.606229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:39.606197 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:31:39.642469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:39.642439 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:41.608716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:41.608680 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:43.208960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:43.208930 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:44.816879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:44.816844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:46.538624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:46.534815 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:47.895344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.895312 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9"] Oct 30 13:31:47.899896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.899871 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.928003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.927969 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9"] Oct 30 13:31:47.935886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.935854 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-552cc3\"/\"kube-root-ca.crt\"" Oct 30 13:31:47.936054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.935883 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-552cc3\"/\"auth-test-model-552cc3-kserve-self-signed-certs\"" Oct 30 13:31:47.936054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.935974 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-552cc3\"/\"openshift-service-ca.crt\"" Oct 30 13:31:47.936153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.936065 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-552cc3\"/\"default-dockercfg-5zv9m\"" Oct 30 13:31:47.982953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.982917 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.983149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.982969 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.983149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.983088 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.983149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.983138 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gt2d\" (UniqueName: \"kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.983323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.983207 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.983365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.983329 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:47.990184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.990160 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768"] Oct 30 13:31:47.995270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:47.995246 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.006249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.006187 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:48.013860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.013801 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-29a3f3-serving-cert\"" Oct 30 13:31:48.017028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.017001 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768"] Oct 30 13:31:48.085324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.085500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085333 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gt2d\" (UniqueName: \"kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.085500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085359 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.085500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085424 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.085500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085481 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.085794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085667 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486fm\" (UniqueName: \"kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.085794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085712 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.085931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085817 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.085931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085857 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.085931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085910 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.086053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085936 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.086053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.085982 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.086217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.086202 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.088009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.087988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.088518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.088502 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.104017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.103991 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gt2d\" (UniqueName: \"kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d\") pod \"auth-test-model-552cc3-kserve-556df44d4f-qcxr9\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.146798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.146704 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:48.187709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.187678 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486fm\" (UniqueName: \"kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.187871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.187788 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.187871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.187833 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.188039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.188011 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.188932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.188903 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.207858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.191336 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.231900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.231770 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:48.238618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.232191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.293777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.291265 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-486fm\" (UniqueName: \"kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm\") pod \"maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.293777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.292812 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:48.319765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.314230 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:48.721546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.721514 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9"] Oct 30 13:31:48.722721 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:31:48.722681 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac6dfe77_6f24_44df_9675_7ded9f7f7dca.slice/crio-9634053acf3998527684d5e25a6ea22718e1a5fb4b2a66822773ff2d02eef9c4 WatchSource:0}: Error finding container 9634053acf3998527684d5e25a6ea22718e1a5fb4b2a66822773ff2d02eef9c4: Status 404 returned error can't find the container with id 9634053acf3998527684d5e25a6ea22718e1a5fb4b2a66822773ff2d02eef9c4 Oct 30 13:31:48.734999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:48.734973 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768"] Oct 30 13:31:48.735797 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:31:48.735774 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b5a289_4aea_4a69_9544_a3326448333a.slice/crio-f048ffc74cf181991c954ecbd6c24c81f50ebb70ef4ce92f30cb3eca56d283d7 WatchSource:0}: Error finding container f048ffc74cf181991c954ecbd6c24c81f50ebb70ef4ce92f30cb3eca56d283d7: Status 404 returned error can't find the container with id f048ffc74cf181991c954ecbd6c24c81f50ebb70ef4ce92f30cb3eca56d283d7 Oct 30 13:31:49.694762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.694726 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" event={"ID":"ac6dfe77-6f24-44df-9675-7ded9f7f7dca","Type":"ContainerStarted","Data":"1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d"} Oct 30 13:31:49.694762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.694770 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" event={"ID":"ac6dfe77-6f24-44df-9675-7ded9f7f7dca","Type":"ContainerStarted","Data":"9634053acf3998527684d5e25a6ea22718e1a5fb4b2a66822773ff2d02eef9c4"} Oct 30 13:31:49.697502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.697476 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" event={"ID":"d1b5a289-4aea-4a69-9544-a3326448333a","Type":"ContainerStarted","Data":"d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a"} Oct 30 13:31:49.697652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.697520 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" event={"ID":"d1b5a289-4aea-4a69-9544-a3326448333a","Type":"ContainerStarted","Data":"f048ffc74cf181991c954ecbd6c24c81f50ebb70ef4ce92f30cb3eca56d283d7"} Oct 30 13:31:49.697652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.697572 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:49.734997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.734943 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" podStartSLOduration=2.7349286189999997 podStartE2EDuration="2.734928619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:49.7330535 +0000 UTC m=+1685.599671674" watchObservedRunningTime="2026-10-30 13:31:49.734928619 +0000 UTC m=+1685.601546792" Oct 30 13:31:49.773721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:49.773665 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" podStartSLOduration=2.349276748 podStartE2EDuration="2.773646251s" totalImagesPullingTime="424.369503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:49.771803417 +0000 UTC m=+1685.638421602" watchObservedRunningTime="2026-10-30 13:31:49.773646251 +0000 UTC m=+1685.640264426" Oct 30 13:31:50.040621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:50.034136 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:50.233013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:50.232975 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:51.816950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:51.816901 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:53.063219 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:31:53.063178 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:31:53.415787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:53.413951 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:54.541231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:54.541185 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:55.103543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:55.103510 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:55.711615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:55.711569 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 13:31:56.643779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:56.643734 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:58.164410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:58.164377 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:58.240109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:58.240071 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:58.290825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:58.290797 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:58.779524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:58.779495 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:31:59.823861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:31:59.823825 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:01.417977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:01.417896 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:03.016088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:03.016045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:04.597927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:04.597889 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:06.304579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:06.304538 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:07.917374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:07.917342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:09.619114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:09.619083 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:09.783631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:09.781826 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:09.920304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:09.920221 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:11.419949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:11.419914 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:12.992434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:12.992400 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:14.624341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:14.624309 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:15.149827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:15.149792 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:16.643103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:16.643049 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:18.270408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:18.270371 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:19.149225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:19.149186 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:19.898622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:19.886671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:21.621076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:21.618908 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:23.058323 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:32:23.058288 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:32:23.311842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:23.311758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:25.001143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:25.001066 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:25.791181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:25.791138 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz"] Oct 30 13:32:25.927777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:25.927732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:26.094099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:26.094012 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:26.268401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:26.268363 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl"] Oct 30 13:32:26.269099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:26.268671 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" podUID="beab246f-c9b1-4346-acbe-49021f319edb" containerName="maas-api" containerID="cri-o://bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60" gracePeriod=30 Oct 30 13:32:26.413283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:26.413254 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:26.974113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:26.974080 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:32:27.089728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.088684 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca\") pod \"beab246f-c9b1-4346-acbe-49021f319edb\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " Oct 30 13:32:27.089728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.088775 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls\") pod \"beab246f-c9b1-4346-acbe-49021f319edb\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " Oct 30 13:32:27.089728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.088926 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv\" (UniqueName: \"kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv\") pod \"beab246f-c9b1-4346-acbe-49021f319edb\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " Oct 30 13:32:27.090108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.089977 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca" pod "beab246f-c9b1-4346-acbe-49021f319edb" (UID: "beab246f-c9b1-4346-acbe-49021f319edb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:32:27.098905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.095222 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv" pod "beab246f-c9b1-4346-acbe-49021f319edb" (UID: "beab246f-c9b1-4346-acbe-49021f319edb"). InnerVolumeSpecName "kube-api-access-sk9kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:32:27.101893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.101859 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls" pod "beab246f-c9b1-4346-acbe-49021f319edb" (UID: "beab246f-c9b1-4346-acbe-49021f319edb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:32:27.111528 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.111497 5693 generic.go:356] "Generic (PLEG): container finished" podID="beab246f-c9b1-4346-acbe-49021f319edb" containerID="bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60" exitCode=0 Oct 30 13:32:27.111842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.111549 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" Oct 30 13:32:27.111948 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.111567 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" event={"ID":"beab246f-c9b1-4346-acbe-49021f319edb","Type":"ContainerDied","Data":"bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60"} Oct 30 13:32:27.112018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.111962 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl" event={"ID":"beab246f-c9b1-4346-acbe-49021f319edb","Type":"ContainerDied","Data":"7b9b2e438f8b25d0ebf4e90119477b2e92a8641c5d95737f8c98125d244bd82f"} Oct 30 13:32:27.112018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.111999 5693 scope.go:122] "RemoveContainer" containerID="bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60" Oct 30 13:32:27.129587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.129567 5693 scope.go:122] "RemoveContainer" containerID="bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60" Oct 30 13:32:27.129976 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:32:27.129948 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60\": container with ID starting with bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60 not found: ID does not exist" containerID="bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60" Oct 30 13:32:27.130038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.129984 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60"} err="failed to get container status \"bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60\": rpc error: code = NotFound desc = could not find container \"bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60\": container with ID starting with bbc885b4e39c41701924e84ff3597e6b53df86a307c814105ffa17dd5e4f6d60 not found: ID does not exist" Oct 30 13:32:27.189890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.189845 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs\") pod \"beab246f-c9b1-4346-acbe-49021f319edb\" (UID: \"beab246f-c9b1-4346-acbe-49021f319edb\") " Oct 30 13:32:27.190814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.190784 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beab246f-c9b1-4346-acbe-49021f319edb-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:32:27.190814 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.190817 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:32:27.191045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.190833 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9kv\" (UniqueName: \"kubernetes.io/projected/beab246f-c9b1-4346-acbe-49021f319edb-kube-api-access-sk9kv\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:32:27.192282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.192255 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs" pod "beab246f-c9b1-4346-acbe-49021f319edb" (UID: "beab246f-c9b1-4346-acbe-49021f319edb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:32:27.335621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.323146 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beab246f-c9b1-4346-acbe-49021f319edb-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:32:27.399446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.397177 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:27.575205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.575163 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl"] Oct 30 13:32:27.652677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:27.646039 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f45f04-9458f684f-2k8rl"] Oct 30 13:32:28.388034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:28.387996 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beab246f-c9b1-4346-acbe-49021f319edb" path="/var/lib/kubelet/pods/beab246f-c9b1-4346-acbe-49021f319edb/volumes" Oct 30 13:32:29.469901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:29.469855 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:29.620682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:29.620649 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:30.792578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:30.792535 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" podUID="33baf68d-9646-4635-92d6-7bc8cc11fba2" containerName="payload-pre-processing" containerID="cri-o://574e66132e5a5c77ccf81ed7e93d5dfdcdcaa1bc3cd21a13e1488c350adbb69b" gracePeriod=25 Oct 30 13:32:31.064315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.064183 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:31.199294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.199261 5693 generic.go:356] "Generic (PLEG): container finished" podID="33baf68d-9646-4635-92d6-7bc8cc11fba2" containerID="574e66132e5a5c77ccf81ed7e93d5dfdcdcaa1bc3cd21a13e1488c350adbb69b" exitCode=0 Oct 30 13:32:31.199460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.199295 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" event={"ID":"33baf68d-9646-4635-92d6-7bc8cc11fba2","Type":"ContainerDied","Data":"574e66132e5a5c77ccf81ed7e93d5dfdcdcaa1bc3cd21a13e1488c350adbb69b"} Oct 30 13:32:31.199460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.199347 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" event={"ID":"33baf68d-9646-4635-92d6-7bc8cc11fba2","Type":"ContainerDied","Data":"cc44ced6bc345b36ecc13edef9797e0ba161028009cfcea7cff80267485a10b8"} Oct 30 13:32:31.199460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.199369 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc44ced6bc345b36ecc13edef9797e0ba161028009cfcea7cff80267485a10b8" Oct 30 13:32:31.199460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.199273 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:32:31.262662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.262632 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h\" (UniqueName: \"kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h\") pod \"33baf68d-9646-4635-92d6-7bc8cc11fba2\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " Oct 30 13:32:31.262957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.262944 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume\") pod \"33baf68d-9646-4635-92d6-7bc8cc11fba2\" (UID: \"33baf68d-9646-4635-92d6-7bc8cc11fba2\") " Oct 30 13:32:31.263355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.263324 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume" pod "33baf68d-9646-4635-92d6-7bc8cc11fba2" (UID: "33baf68d-9646-4635-92d6-7bc8cc11fba2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:32:31.263507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.263498 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33baf68d-9646-4635-92d6-7bc8cc11fba2-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:32:31.265009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.264972 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h" pod "33baf68d-9646-4635-92d6-7bc8cc11fba2" (UID: "33baf68d-9646-4635-92d6-7bc8cc11fba2"). InnerVolumeSpecName "kube-api-access-5z22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:32:31.364775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:31.364684 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z22h\" (UniqueName: \"kubernetes.io/projected/33baf68d-9646-4635-92d6-7bc8cc11fba2-kube-api-access-5z22h\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:32:32.207203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:32.207172 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz" Oct 30 13:32:32.258413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:32.258383 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz"] Oct 30 13:32:32.269967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:32.269941 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f45f04-d7cdb4695-zf8hz"] Oct 30 13:32:32.680614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:32.680570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:34.256062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:34.256028 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33baf68d-9646-4635-92d6-7bc8cc11fba2" path="/var/lib/kubelet/pods/33baf68d-9646-4635-92d6-7bc8cc11fba2/volumes" Oct 30 13:32:34.391213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:34.391175 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:36.121055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:36.121017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:37.753186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:37.753151 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:39.498963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:39.498928 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:39.872990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:39.872905 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:41.172589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:41.172553 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:42.773730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:42.773695 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:44.347859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:44.347821 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:46.024205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.024171 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:46.757062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757032 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88"] Oct 30 13:32:46.757749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757735 5693 cpu_manager.go:414] "containerMap: removing container" podUID="beab246f-c9b1-4346-acbe-49021f319edb" containerName="maas-api" Oct 30 13:32:46.757820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757750 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beab246f-c9b1-4346-acbe-49021f319edb" containerName="maas-api" Oct 30 13:32:46.757820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757780 5693 cpu_manager.go:414] "containerMap: removing container" podUID="33baf68d-9646-4635-92d6-7bc8cc11fba2" containerName="payload-pre-processing" Oct 30 13:32:46.757820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757785 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33baf68d-9646-4635-92d6-7bc8cc11fba2" containerName="payload-pre-processing" Oct 30 13:32:46.757930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757907 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="33baf68d-9646-4635-92d6-7bc8cc11fba2" containerName="payload-pre-processing" Oct 30 13:32:46.757930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.757922 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="beab246f-c9b1-4346-acbe-49021f319edb" containerName="maas-api" Oct 30 13:32:46.761838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.761813 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.784282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.784255 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88"] Oct 30 13:32:46.792067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.792036 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-167965-openshift-default-dockercfg-rfp6j\"" Oct 30 13:32:46.843139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843106 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gw25\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843252 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843302 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843421 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843447 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843557 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843633 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843654 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.843742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.843731 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.944903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.944868 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.944950 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.944975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945044 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945127 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gw25\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945257 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945369 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945387 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945399 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945445 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945764 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945825 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.945897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.945839 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.947557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.947532 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.947776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.947758 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.964460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.964429 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:46.964569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:46.964504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gw25\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:47.076636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:47.076543 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:47.406708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:47.406679 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88"] Oct 30 13:32:47.407902 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:32:47.407877 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f94fbb_1bf6_4a48_8ac9_0fcdb78cfc99.slice/crio-02ef70a05801dfc41bd5a78822497f14abc10a7c83dce9af6686bba3789b3eac WatchSource:0}: Error finding container 02ef70a05801dfc41bd5a78822497f14abc10a7c83dce9af6686bba3789b3eac: Status 404 returned error can't find the container with id 02ef70a05801dfc41bd5a78822497f14abc10a7c83dce9af6686bba3789b3eac Oct 30 13:32:47.411066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:47.411023 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:32:47.411164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:47.411114 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:32:47.411164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:47.411154 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:32:47.750338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:47.750297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:48.328888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:48.328852 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" event={"ID":"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99","Type":"ContainerStarted","Data":"551a53ad7e69765bc326997897b7422568cfa9bcbad99aab11170b7cde27883b"} Oct 30 13:32:48.329372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:48.328920 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" event={"ID":"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99","Type":"ContainerStarted","Data":"02ef70a05801dfc41bd5a78822497f14abc10a7c83dce9af6686bba3789b3eac"} Oct 30 13:32:48.396351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:48.396305 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podStartSLOduration=2.396291923 podStartE2EDuration="2.396291923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:48.380977588 +0000 UTC m=+1744.247595761" watchObservedRunningTime="2026-10-30 13:32:48.396291923 +0000 UTC m=+1744.262910097" Oct 30 13:32:48.444029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:48.443998 5693 scope.go:122] "RemoveContainer" containerID="f98c287b4ca54380ba8ff871532d333b03217ecfbb6ebf2e4029023aeef1deeb" Oct 30 13:32:49.077750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:49.077718 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:49.078926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:49.078901 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:49.079056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:49.078946 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:49.309775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:49.309741 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:50.077758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:50.077725 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:50.078170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:50.077776 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:51.076970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:51.076935 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:51.077181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:51.077000 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:51.201083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:51.201040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:51.370793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:51.369134 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:51.819817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:51.819773 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:52.077074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:52.076984 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:52.077231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:52.077068 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:53.077110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:53.077074 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:53.077652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:53.077144 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:53.077652 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:32:53.077560 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:32:53.206620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:53.202517 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:54.077146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:54.077110 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:54.077553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:54.077182 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:54.671287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:54.671236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:54.962895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:54.948110 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:55.077680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:55.077645 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:32:55.078177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:55.077705 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 13:32:56.082771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:56.082743 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:56.083225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:56.083211 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:56.084100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:56.084079 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:32:56.736658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:56.736526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.425782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:58.425742 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.723443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:58.723360 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.855819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:58.855784 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.981498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:58.981467 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d"] Oct 30 13:32:58.981735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:58.981714 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" containerID="cri-o://5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85" gracePeriod=30 Oct 30 13:32:59.182631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:59.177689 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:59.490384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:59.490355 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:59.711199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:32:59.711151 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:00.081997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.081954 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2"] Oct 30 13:33:00.087777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.087746 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.110548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.110520 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2"] Oct 30 13:33:00.117694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.117657 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-143cf2-openshift-default-dockercfg-g65j8\"" Oct 30 13:33:00.167488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.167460 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.205581 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.205755 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.205788 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.205863 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.205982 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.206013 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzmm\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.206043 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.206114 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.206906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.206233 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.321040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.321000 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzmm\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.322482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.322451 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.323643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.323622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.324541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.324520 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.324764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.324744 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.325089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.325068 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.326521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.326504 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.327094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.327073 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.327625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.324355 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.327760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.325439 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.327866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.323467 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.336059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.327484 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.336209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.335975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.337022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.336994 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.339135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.338470 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.339298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.338859 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:00.362618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.350045 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzmm\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.363864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.363830 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.366967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.366902 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token\") pod \"e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.423410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.423094 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:00.505323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.505286 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:00.963776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.963747 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2"] Oct 30 13:33:00.965483 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:33:00.965456 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91110e47_3f71_4d75_bccd_bf1c5455b5e3.slice/crio-b2944dcc566ce3c84465e1c4355c23c33af5002d0f7358bea8493297a77fb115 WatchSource:0}: Error finding container b2944dcc566ce3c84465e1c4355c23c33af5002d0f7358bea8493297a77fb115: Status 404 returned error can't find the container with id b2944dcc566ce3c84465e1c4355c23c33af5002d0f7358bea8493297a77fb115 Oct 30 13:33:00.968189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.968157 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:33:00.968261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.968226 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:33:00.968261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:00.968258 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:33:01.541302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:01.541261 5693 patch_prober.go:29] interesting pod/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:33:01.541781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:01.541313 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:33:01.552536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:01.552498 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" event={"ID":"91110e47-3f71-4d75-bccd-bf1c5455b5e3","Type":"ContainerStarted","Data":"5647050206aedbdb0ec64ad2a7613cfec15a3ea5efb7ae93f36ebebca1834da6"} Oct 30 13:33:01.552699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:01.552558 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" event={"ID":"91110e47-3f71-4d75-bccd-bf1c5455b5e3","Type":"ContainerStarted","Data":"b2944dcc566ce3c84465e1c4355c23c33af5002d0f7358bea8493297a77fb115"} Oct 30 13:33:01.600656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:01.597289 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podStartSLOduration=1.597272146 podStartE2EDuration="1.597272146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:01.595756458 +0000 UTC m=+1757.462374668" watchObservedRunningTime="2026-10-30 13:33:01.597272146 +0000 UTC m=+1757.463890320" Oct 30 13:33:02.071754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:02.071689 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:02.424631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:02.424530 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:02.430306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:02.430278 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:02.559434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:02.559398 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:02.560408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:02.560388 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:33:03.719149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:03.719113 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:04.075114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.075082 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:04.214021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.213984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:04.300652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.300628 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:33:04.399708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.399672 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.399880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.399743 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.399967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.399883 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.400037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.399998 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.400037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400001 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:04.400157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400070 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.400219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400159 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:04.400274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400223 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:04.400274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400227 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.400381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400296 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.400381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400334 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.400531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400491 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:04.401227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.400905 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data\") pod \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\" (UID: \"61a4c160-ebaf-40a4-a5b5-f508d2aaba50\") " Oct 30 13:33:04.401227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.401173 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:04.401805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.401778 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.401914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.401809 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.401914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.401825 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.401914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.401838 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.401914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.401850 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.403136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.403105 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:33:04.403253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.403181 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:33:04.403253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.403219 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "kube-api-access-mlwkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:04.404955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.404931 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token" pod "61a4c160-ebaf-40a4-a5b5-f508d2aaba50" (UID: "61a4c160-ebaf-40a4-a5b5-f508d2aaba50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:04.503570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.503533 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.503570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.503564 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.503570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.503572 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.503806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.503583 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwkq\" (UniqueName: \"kubernetes.io/projected/61a4c160-ebaf-40a4-a5b5-f508d2aaba50-kube-api-access-mlwkq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:33:04.576028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.575945 5693 generic.go:356] "Generic (PLEG): container finished" podID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerID="5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85" exitCode=0 Oct 30 13:33:04.576028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.575989 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" Oct 30 13:33:04.576028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.576016 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" event={"ID":"61a4c160-ebaf-40a4-a5b5-f508d2aaba50","Type":"ContainerDied","Data":"5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85"} Oct 30 13:33:04.576296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.576068 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d" event={"ID":"61a4c160-ebaf-40a4-a5b5-f508d2aaba50","Type":"ContainerDied","Data":"5b7e909fd5212afdb0937a7bfced0706e1ed135fc875e3244061c102c746310d"} Oct 30 13:33:04.576296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.576092 5693 scope.go:122] "RemoveContainer" containerID="5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85" Oct 30 13:33:04.592722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.592703 5693 scope.go:122] "RemoveContainer" containerID="d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a" Oct 30 13:33:04.603572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.603548 5693 scope.go:122] "RemoveContainer" containerID="5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85" Oct 30 13:33:04.603905 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:33:04.603880 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85\": container with ID starting with 5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85 not found: ID does not exist" containerID="5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85" Oct 30 13:33:04.603969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.603915 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85"} err="failed to get container status \"5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85\": rpc error: code = NotFound desc = could not find container \"5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85\": container with ID starting with 5c38421075da101f858a7629bd4e5097d4fc38a60a852920de2d49132d54ef85 not found: ID does not exist" Oct 30 13:33:04.603969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.603936 5693 scope.go:122] "RemoveContainer" containerID="d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a" Oct 30 13:33:04.604200 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:33:04.604180 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a\": container with ID starting with d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a not found: ID does not exist" containerID="d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a" Oct 30 13:33:04.604270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.604207 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a"} err="failed to get container status \"d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a\": rpc error: code = NotFound desc = could not find container \"d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a\": container with ID starting with d4ab6f2858e9922d09898d65c521e6b2c359b4445b85d00768e1a768d06f711a not found: ID does not exist" Oct 30 13:33:04.628970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.628926 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d"] Oct 30 13:33:04.632249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:04.632222 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f45f04-openshift-default-7bbc5454d5-dpm8d"] Oct 30 13:33:05.413048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:05.413013 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:05.720001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:05.719924 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:05.868231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:05.868196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:06.262344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:06.262312 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" path="/var/lib/kubelet/pods/61a4c160-ebaf-40a4-a5b5-f508d2aaba50/volumes" Oct 30 13:33:07.307235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:07.307201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:09.036628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:09.032282 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:10.589420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:10.589304 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:12.189828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:12.189785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:13.840388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:13.840333 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:15.437652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:15.437185 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:16.958407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:16.958370 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:18.658622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:18.652795 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:20.316111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:20.316076 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:21.980621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:21.974342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:23.056038 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:33:23.055994 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:33:23.575621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:23.571497 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:25.351061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:25.351025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:26.970036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.970000 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt"] Oct 30 13:33:26.970773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.970758 5693 cpu_manager.go:414] "containerMap: removing container" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" Oct 30 13:33:26.970824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.970774 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" Oct 30 13:33:26.970824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.970790 5693 cpu_manager.go:414] "containerMap: removing container" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" Oct 30 13:33:26.970824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.970796 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" Oct 30 13:33:26.970924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.970897 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" Oct 30 13:33:26.976044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:26.976024 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.003841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.003809 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-143cf2-serving-cert\"" Oct 30 13:33:27.007657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.007589 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt"] Oct 30 13:33:27.102077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.102035 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.103500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.103475 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.103675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.103617 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.103741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.103726 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9pfl\" (UniqueName: \"kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.103780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.103766 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.213910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.213871 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.214914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.214885 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.215255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.215230 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9pfl\" (UniqueName: \"kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.215429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.215412 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.238292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.221537 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.244126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.244097 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.249504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.248578 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.252144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.252084 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.258634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.258591 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9pfl\" (UniqueName: \"kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl\") pod \"maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.290146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.290115 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:27.393355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.393319 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.593277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.593243 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt"] Oct 30 13:33:27.595283 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:33:27.595251 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31c2a89_ef80_463f_8150_2cd905423bcd.slice/crio-d595642fc6e87f88ee4cdd1616da32aba807bc05fc4f12997f6052ed7ad84e03 WatchSource:0}: Error finding container d595642fc6e87f88ee4cdd1616da32aba807bc05fc4f12997f6052ed7ad84e03: Status 404 returned error can't find the container with id d595642fc6e87f88ee4cdd1616da32aba807bc05fc4f12997f6052ed7ad84e03 Oct 30 13:33:27.799040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:27.799000 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" event={"ID":"f31c2a89-ef80-463f-8150-2cd905423bcd","Type":"ContainerStarted","Data":"d595642fc6e87f88ee4cdd1616da32aba807bc05fc4f12997f6052ed7ad84e03"} Oct 30 13:33:28.834854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:28.834818 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" event={"ID":"f31c2a89-ef80-463f-8150-2cd905423bcd","Type":"ContainerStarted","Data":"e1328e58bdb21a2b42f2a2b99f31f8a867e7143b965ce4c65318ed201aa96ef3"} Oct 30 13:33:28.835337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:28.835017 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:28.876437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:28.876370 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" podStartSLOduration=2.426836801 podStartE2EDuration="2.876350805s" totalImagesPullingTime="449.514004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:28.873772707 +0000 UTC m=+1784.740390884" watchObservedRunningTime="2026-10-30 13:33:28.876350805 +0000 UTC m=+1784.742968979" Oct 30 13:33:29.004821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:29.004768 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:31.056629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:31.056566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:32.546619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:32.540935 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:34.210123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:34.210075 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:34.847708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:34.847678 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:33:35.893043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:35.892960 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:37.544181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:37.544146 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:39.214118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:39.214083 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:40.745776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:40.745675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:42.413308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:42.413274 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:44.028620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:44.028566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:45.805304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:45.805271 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:47.729851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:47.729764 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:49.542690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:49.536382 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:51.233062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:51.233026 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:52.947688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:52.936222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:53.056836 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:33:53.056802 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:33:54.816963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:54.816929 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:54.966968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:54.966880 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:56.714836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:56.714800 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:58.383564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:33:58.383529 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:00.107897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:00.107865 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:00.276357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:00.276321 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:01.815377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:01.815339 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:02.384220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:02.384188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:03.591216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:03.591176 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:03.616668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:03.615719 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:34:03.616668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:03.615793 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:34:05.286453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.286419 5693 generic.go:356] "Generic (PLEG): container finished" podID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerID="5647050206aedbdb0ec64ad2a7613cfec15a3ea5efb7ae93f36ebebca1834da6" exitCode=0 Oct 30 13:34:05.287101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.286497 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" event={"ID":"91110e47-3f71-4d75-bccd-bf1c5455b5e3","Type":"ContainerDied","Data":"5647050206aedbdb0ec64ad2a7613cfec15a3ea5efb7ae93f36ebebca1834da6"} Oct 30 13:34:05.287194 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.287117 5693 scope.go:122] "RemoveContainer" containerID="5647050206aedbdb0ec64ad2a7613cfec15a3ea5efb7ae93f36ebebca1834da6" Oct 30 13:34:05.288928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.288897 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:34:05.289036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.288974 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:34:05.289036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.289011 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:34:05.414136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.414099 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:05.582466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:05.582383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:06.296536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:06.296498 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" event={"ID":"91110e47-3f71-4d75-bccd-bf1c5455b5e3","Type":"ContainerStarted","Data":"ed5692a8c1241b47df9fdf2377b5ae25e2e840a43aebc0d583052c3a5e9fbe0b"} Oct 30 13:34:06.424613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:06.424564 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:34:06.424809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:06.424757 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 13:34:06.424889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:06.424814 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 13:34:07.107707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:07.107419 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:07.424996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:07.424909 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 13:34:07.424996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:07.424959 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 13:34:08.424731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:08.424693 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 13:34:08.424909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:08.424747 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 13:34:08.800977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:08.800933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:09.424895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:09.424859 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 13:34:09.425086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:09.424913 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 13:34:10.424471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:10.424438 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 13:34:10.424882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:10.424492 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 13:34:10.518350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:10.517730 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:11.424134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:11.424100 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 13:34:11.424316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:11.424153 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 13:34:12.218508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:12.218471 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:12.428229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:12.428198 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:34:12.428637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:12.428615 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:34:12.429262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:12.429245 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:34:13.930215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:13.930182 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:15.663619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:15.660137 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:17.412629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:17.412560 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:19.213194 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:19.213146 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:20.811932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:20.811897 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:22.539493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:22.539457 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:23.056760 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:23.056724 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:34:24.132895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:24.132856 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:25.735929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:25.735888 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:26.707644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.704313 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:26.900542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.900511 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf"] Oct 30 13:34:26.901320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.901290 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" containerID="cri-o://af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475" gracePeriod=30 Oct 30 13:34:26.912439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.912412 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7"] Oct 30 13:34:26.912884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.912852 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" containerID="cri-o://5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b" gracePeriod=30 Oct 30 13:34:26.918871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.917434 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd"] Oct 30 13:34:26.918871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:26.917637 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" containerID="cri-o://ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273" gracePeriod=30 Oct 30 13:34:27.505868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:27.505830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:29.247417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:29.245891 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:31.049662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:31.049622 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:32.223552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.223520 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:34:32.295645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.289743 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:34:32.295645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.294374 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:34:32.366133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366093 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366148 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.366310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366178 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.366310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.366310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366275 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.366519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366365 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.366519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366505 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.366619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366556 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.366619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366557 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366648 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366731 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366773 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366812 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366904 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.366976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.366934 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.367037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.367023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.367170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.367154 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs\") pod \"239559a4-f5c0-4b59-85d3-3cd052a548de\" (UID: \"239559a4-f5c0-4b59-85d3-3cd052a548de\") " Oct 30 13:34:32.367212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.367197 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.367290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.367276 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket\") pod \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\" (UID: \"edafbadc-9d8c-4547-ba9d-b721c3b254a5\") " Oct 30 13:34:32.368275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.368253 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.369211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.369180 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.369519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.369483 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.369665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.369514 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:32.369745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.369719 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.370261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.370234 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.370518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.370492 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.370644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.370617 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.370798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.370773 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:32.370905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.370813 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.371879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.371852 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "kube-api-access-sffbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:32.372133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.372109 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.372435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.372202 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:32.373170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.373148 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:32.373273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.373167 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "kube-api-access-z79k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:32.374041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.374015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token" pod "239559a4-f5c0-4b59-85d3-3cd052a548de" (UID: "239559a4-f5c0-4b59-85d3-3cd052a548de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:32.374473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.374449 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:32.374582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.374477 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy" pod "edafbadc-9d8c-4547-ba9d-b721c3b254a5" (UID: "edafbadc-9d8c-4547-ba9d-b721c3b254a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.469384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469295 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.469384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469358 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.469651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469439 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.469651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469482 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.469651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469561 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.469851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469648 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.469851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469665 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.469851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469826 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.469851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469830 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.470071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469880 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.470071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469889 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.470071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.470071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.469890 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:32.470071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470024 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj\") pod \"18a1b04f-20d2-4b26-961f-05bf8646844b\" (UID: \"18a1b04f-20d2-4b26-961f-05bf8646844b\") " Oct 30 13:34:32.470885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470855 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.470885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470889 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470904 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470916 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470930 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1b04f-20d2-4b26-961f-05bf8646844b-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470944 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470958 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79k5\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-kube-api-access-z79k5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470972 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470986 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.470999 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471013 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471026 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471038 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471051 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239559a4-f5c0-4b59-85d3-3cd052a548de-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471065 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471078 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471091 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471106 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sffbh\" (UniqueName: \"kubernetes.io/projected/239559a4-f5c0-4b59-85d3-3cd052a548de-kube-api-access-sffbh\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471134 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239559a4-f5c0-4b59-85d3-3cd052a548de-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471150 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471162 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edafbadc-9d8c-4547-ba9d-b721c3b254a5-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.471920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.471175 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.472213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.472046 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:32.472213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.472091 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:32.472213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.472178 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:32.472375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.472356 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj" pod "18a1b04f-20d2-4b26-961f-05bf8646844b" (UID: "18a1b04f-20d2-4b26-961f-05bf8646844b"). InnerVolumeSpecName "kube-api-access-wrkgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:32.512700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.512665 5693 generic.go:356] "Generic (PLEG): container finished" podID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerID="af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475" exitCode=0 Oct 30 13:34:32.512905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.512728 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" Oct 30 13:34:32.512905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.512746 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" event={"ID":"239559a4-f5c0-4b59-85d3-3cd052a548de","Type":"ContainerDied","Data":"af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475"} Oct 30 13:34:32.512905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.512836 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf" event={"ID":"239559a4-f5c0-4b59-85d3-3cd052a548de","Type":"ContainerDied","Data":"3ed89f77f4d3ec0e7ad7d6b8b6a2fd7980c6495c39bfd391145391176ba4b721"} Oct 30 13:34:32.512905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.512867 5693 scope.go:122] "RemoveContainer" containerID="af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475" Oct 30 13:34:32.514558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.514534 5693 generic.go:356] "Generic (PLEG): container finished" podID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerID="ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273" exitCode=0 Oct 30 13:34:32.514791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.514757 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" event={"ID":"18a1b04f-20d2-4b26-961f-05bf8646844b","Type":"ContainerDied","Data":"ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273"} Oct 30 13:34:32.514912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.514806 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" Oct 30 13:34:32.514912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.514820 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd" event={"ID":"18a1b04f-20d2-4b26-961f-05bf8646844b","Type":"ContainerDied","Data":"a09829f43bbaaf2b7e9d2b8f4e3c5e1a1739ada463ee8590778ce6fb456b7cbf"} Oct 30 13:34:32.521972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.521945 5693 generic.go:356] "Generic (PLEG): container finished" podID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerID="5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b" exitCode=0 Oct 30 13:34:32.522114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.522005 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" event={"ID":"edafbadc-9d8c-4547-ba9d-b721c3b254a5","Type":"ContainerDied","Data":"5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b"} Oct 30 13:34:32.522114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.522074 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" event={"ID":"edafbadc-9d8c-4547-ba9d-b721c3b254a5","Type":"ContainerDied","Data":"5d998f6f068a84125c80188632991f14b2cf78dada452d5d5ba06ffe42cd67b0"} Oct 30 13:34:32.522238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.522158 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7" Oct 30 13:34:32.527786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.527763 5693 scope.go:122] "RemoveContainer" containerID="af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475" Oct 30 13:34:32.528277 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:32.528247 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475\": container with ID starting with af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475 not found: ID does not exist" containerID="af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475" Oct 30 13:34:32.528378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.528284 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475"} err="failed to get container status \"af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475\": rpc error: code = NotFound desc = could not find container \"af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475\": container with ID starting with af448d1502b4b7041071c53e351bbb229d5f0850f38cd0904d882934ef6c3475 not found: ID does not exist" Oct 30 13:34:32.528378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.528343 5693 scope.go:122] "RemoveContainer" containerID="ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273" Oct 30 13:34:32.540022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.540002 5693 scope.go:122] "RemoveContainer" containerID="ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273" Oct 30 13:34:32.540350 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:32.540331 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273\": container with ID starting with ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273 not found: ID does not exist" containerID="ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273" Oct 30 13:34:32.540398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.540358 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273"} err="failed to get container status \"ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273\": rpc error: code = NotFound desc = could not find container \"ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273\": container with ID starting with ef4c83268c2bd37f7bfac880d5c502cc5745a8dd38721f1a0bb109cbff537273 not found: ID does not exist" Oct 30 13:34:32.540398 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.540375 5693 scope.go:122] "RemoveContainer" containerID="5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b" Oct 30 13:34:32.551121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.551097 5693 scope.go:122] "RemoveContainer" containerID="5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b" Oct 30 13:34:32.551424 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:32.551401 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b\": container with ID starting with 5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b not found: ID does not exist" containerID="5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b" Oct 30 13:34:32.551530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.551431 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b"} err="failed to get container status \"5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b\": rpc error: code = NotFound desc = could not find container \"5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b\": container with ID starting with 5646acd3ee4e8a97e3f0c11c4545f125665e8124986f902ff2f84be234d7e57b not found: ID does not exist" Oct 30 13:34:32.573116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.573087 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.573116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.573113 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.573116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.573122 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.573357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.573130 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkgj\" (UniqueName: \"kubernetes.io/projected/18a1b04f-20d2-4b26-961f-05bf8646844b-kube-api-access-wrkgj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:32.582670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.582639 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd"] Oct 30 13:34:32.587075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.587046 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-nnjcd"] Oct 30 13:34:32.621328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.621294 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf"] Oct 30 13:34:32.625283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.625260 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-rdvrf"] Oct 30 13:34:32.659166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.659138 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7"] Oct 30 13:34:32.665489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.665463 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-khng7"] Oct 30 13:34:32.837250 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:32.832850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:34.256536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:34.256502 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" path="/var/lib/kubelet/pods/18a1b04f-20d2-4b26-961f-05bf8646844b/volumes" Oct 30 13:34:34.257101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:34.257086 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" path="/var/lib/kubelet/pods/239559a4-f5c0-4b59-85d3-3cd052a548de/volumes" Oct 30 13:34:34.257547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:34.257533 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" path="/var/lib/kubelet/pods/edafbadc-9d8c-4547-ba9d-b721c3b254a5/volumes" Oct 30 13:34:34.604042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:34.603957 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:36.219827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:36.219726 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:37.533913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:37.533879 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:37.679519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:37.679477 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:38.001996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.001959 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:38.944424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.944391 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df"] Oct 30 13:34:38.944975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.944628 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" containerID="cri-o://49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674" gracePeriod=30 Oct 30 13:34:38.956869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.956840 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw"] Oct 30 13:34:38.957096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.957073 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" containerID="cri-o://1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02" gracePeriod=30 Oct 30 13:34:38.961554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.961525 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6"] Oct 30 13:34:38.961751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:38.961726 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" containerID="cri-o://622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607" gracePeriod=30 Oct 30 13:34:39.673761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:39.673706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:41.268287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:41.268207 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:42.906098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:42.906062 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:44.173153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.170484 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt"] Oct 30 13:34:44.173153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.170745 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" podUID="f31c2a89-ef80-463f-8150-2cd905423bcd" containerName="maas-api" containerID="cri-o://e1328e58bdb21a2b42f2a2b99f31f8a867e7143b965ce4c65318ed201aa96ef3" gracePeriod=30 Oct 30 13:34:44.264539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.264427 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:44.341346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.341315 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:34:44.343353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.343309 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:34:44.366254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.366233 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:34:44.378081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.377989 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.378335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378315 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.378437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378364 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.378437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378423 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.378558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378485 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.378665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378564 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.378665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378575 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.378783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378662 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.378783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378725 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.378889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378799 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.378889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378827 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.378889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378842 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.378889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.378853 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.379100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379019 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.379100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379059 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.379216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379138 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.379216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379196 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.379326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379220 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.379326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379254 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data\") pod \"f533284d-9658-4d62-9502-43b0d7e9ce9b\" (UID: \"f533284d-9658-4d62-9502-43b0d7e9ce9b\") " Oct 30 13:34:44.379326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379290 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy\") pod \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\" (UID: \"af5f7f81-8d55-42c6-8ff6-718e868da9d6\") " Oct 30 13:34:44.379781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.379587 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.380283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.380075 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.380283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.380102 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.380283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.380119 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.380283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.380076 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:44.380561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.380328 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.380959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.380689 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.381123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.381098 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.381345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.381322 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:44.381832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.381808 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:44.382218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.381988 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:44.382218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.382009 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "kube-api-access-5dfrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:44.382218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.382066 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:44.382218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.382189 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.382386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.382296 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.382928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.382902 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "kube-api-access-7c95d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:44.383021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.382915 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.383378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.383342 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo" pod "af5f7f81-8d55-42c6-8ff6-718e868da9d6" (UID: "af5f7f81-8d55-42c6-8ff6-718e868da9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:44.386047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.386001 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy" pod "f533284d-9658-4d62-9502-43b0d7e9ce9b" (UID: "f533284d-9658-4d62-9502-43b0d7e9ce9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538266 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f533284d-9658-4d62-9502-43b0d7e9ce9b-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538308 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538323 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538338 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c95d\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-kube-api-access-7c95d\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538354 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538366 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538379 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538392 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538404 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538416 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538430 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5f7f81-8d55-42c6-8ff6-718e868da9d6-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538441 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5f7f81-8d55-42c6-8ff6-718e868da9d6-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538454 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dfrj\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-kube-api-access-5dfrj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538466 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.547186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.538479 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f533284d-9658-4d62-9502-43b0d7e9ce9b-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.627166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.627121 5693 generic.go:356] "Generic (PLEG): container finished" podID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerID="1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02" exitCode=0 Oct 30 13:34:44.627505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.627483 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" event={"ID":"af5f7f81-8d55-42c6-8ff6-718e868da9d6","Type":"ContainerDied","Data":"1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02"} Oct 30 13:34:44.627677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.627665 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" event={"ID":"af5f7f81-8d55-42c6-8ff6-718e868da9d6","Type":"ContainerDied","Data":"0813c53dfdb6b96d0c1ea05257c06eaab849e2e53dc9e658d7da03f19783b280"} Oct 30 13:34:44.627783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.627773 5693 scope.go:122] "RemoveContainer" containerID="1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02" Oct 30 13:34:44.628003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.627990 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw" Oct 30 13:34:44.645117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.645083 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.645397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.645374 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.645666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.645642 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.645932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.645910 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.646189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.646093 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.679624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.646462 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.719720 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.719776 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.719909 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket\") pod \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\" (UID: \"4f3a03f2-e04a-4c64-af75-902820c8ff2b\") " Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.704453 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.704309 5693 generic.go:356] "Generic (PLEG): container finished" podID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerID="49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674" exitCode=0 Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.691153 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.693223 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.685460 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.725368 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.704341 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" event={"ID":"f533284d-9658-4d62-9502-43b0d7e9ce9b","Type":"ContainerDied","Data":"49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674"} Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.725500 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df" event={"ID":"f533284d-9658-4d62-9502-43b0d7e9ce9b","Type":"ContainerDied","Data":"1d14bf3bdef19a41d0bde8c4ee71d05b7aaae5307efa3a7642f14394cb6ffb3f"} Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.726741 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.758787 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.759009 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "kube-api-access-5k4c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.774816 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.775774 5693 generic.go:356] "Generic (PLEG): container finished" podID="f31c2a89-ef80-463f-8150-2cd905423bcd" containerID="e1328e58bdb21a2b42f2a2b99f31f8a867e7143b965ce4c65318ed201aa96ef3" exitCode=0 Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.775945 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" event={"ID":"f31c2a89-ef80-463f-8150-2cd905423bcd","Type":"ContainerDied","Data":"e1328e58bdb21a2b42f2a2b99f31f8a867e7143b965ce4c65318ed201aa96ef3"} Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.779366 5693 generic.go:356] "Generic (PLEG): container finished" podID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerID="622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607" exitCode=0 Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.779413 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" event={"ID":"4f3a03f2-e04a-4c64-af75-902820c8ff2b","Type":"ContainerDied","Data":"622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607"} Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.779462 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" event={"ID":"4f3a03f2-e04a-4c64-af75-902820c8ff2b","Type":"ContainerDied","Data":"00d4c70e00192e4f498204e852ebd02c01a94291ac1595c4ad1bd7f9bdad6712"} Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.779795 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo" pod "4f3a03f2-e04a-4c64-af75-902820c8ff2b" (UID: "4f3a03f2-e04a-4c64-af75-902820c8ff2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824560 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k4c8\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-kube-api-access-5k4c8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824608 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824635 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824649 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824662 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824679 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824696 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824710 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.824723 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a03f2-e04a-4c64-af75-902820c8ff2b-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:44.846947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.830551 5693 scope.go:122] "RemoveContainer" containerID="1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02" Oct 30 13:34:44.846947 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:44.833688 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02\": container with ID starting with 1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02 not found: ID does not exist" containerID="1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02" Oct 30 13:34:44.846947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.833724 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02"} err="failed to get container status \"1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02\": rpc error: code = NotFound desc = could not find container \"1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02\": container with ID starting with 1fd98caf52ff9bc0a0cf32944824896636ca7d0fe9806e87d431891cf4f41f02 not found: ID does not exist" Oct 30 13:34:44.846947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.833743 5693 scope.go:122] "RemoveContainer" containerID="49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674" Oct 30 13:34:44.846947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.841153 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:34:44.846947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.841234 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6" Oct 30 13:34:44.864815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.864770 5693 scope.go:122] "RemoveContainer" containerID="49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674" Oct 30 13:34:44.865186 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:44.865154 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674\": container with ID starting with 49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674 not found: ID does not exist" containerID="49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674" Oct 30 13:34:44.865288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.865199 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674"} err="failed to get container status \"49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674\": rpc error: code = NotFound desc = could not find container \"49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674\": container with ID starting with 49d5997fd57ced0802bf73fca439c7f1e68d16c2ebf8b8a15aa9f06e1e927674 not found: ID does not exist" Oct 30 13:34:44.865288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.865226 5693 scope.go:122] "RemoveContainer" containerID="622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607" Oct 30 13:34:44.883388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.883353 5693 scope.go:122] "RemoveContainer" containerID="622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607" Oct 30 13:34:44.883761 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:44.883733 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607\": container with ID starting with 622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607 not found: ID does not exist" containerID="622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607" Oct 30 13:34:44.883871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.883771 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607"} err="failed to get container status \"622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607\": rpc error: code = NotFound desc = could not find container \"622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607\": container with ID starting with 622c7b4eb692322dab0b8157124d6db20b7eb908645cd825de5c613213c35607 not found: ID does not exist" Oct 30 13:34:44.953246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.953191 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw"] Oct 30 13:34:44.955990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.955966 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9klxw"] Oct 30 13:34:44.989781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.989754 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:44.991044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.991007 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6"] Oct 30 13:34:44.993362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:44.993340 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rnzs6"] Oct 30 13:34:45.027592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.027565 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls\") pod \"f31c2a89-ef80-463f-8150-2cd905423bcd\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " Oct 30 13:34:45.027744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.027612 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df"] Oct 30 13:34:45.027744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.027677 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca\") pod \"f31c2a89-ef80-463f-8150-2cd905423bcd\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " Oct 30 13:34:45.027820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.027751 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs\") pod \"f31c2a89-ef80-463f-8150-2cd905423bcd\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " Oct 30 13:34:45.027856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.027845 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl\" (UniqueName: \"kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl\") pod \"f31c2a89-ef80-463f-8150-2cd905423bcd\" (UID: \"f31c2a89-ef80-463f-8150-2cd905423bcd\") " Oct 30 13:34:45.028029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.028006 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca" pod "f31c2a89-ef80-463f-8150-2cd905423bcd" (UID: "f31c2a89-ef80-463f-8150-2cd905423bcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:45.028459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.028435 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31c2a89-ef80-463f-8150-2cd905423bcd-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:45.030003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.029978 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls" pod "f31c2a89-ef80-463f-8150-2cd905423bcd" (UID: "f31c2a89-ef80-463f-8150-2cd905423bcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:45.030139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.030004 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs" pod "f31c2a89-ef80-463f-8150-2cd905423bcd" (UID: "f31c2a89-ef80-463f-8150-2cd905423bcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:45.032315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.032294 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl" pod "f31c2a89-ef80-463f-8150-2cd905423bcd" (UID: "f31c2a89-ef80-463f-8150-2cd905423bcd"). InnerVolumeSpecName "kube-api-access-x9pfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:45.034433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.034407 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-n42df"] Oct 30 13:34:45.130512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.130435 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:45.130512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.130464 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31c2a89-ef80-463f-8150-2cd905423bcd-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:45.130512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.130474 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9pfl\" (UniqueName: \"kubernetes.io/projected/f31c2a89-ef80-463f-8150-2cd905423bcd-kube-api-access-x9pfl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:45.146594 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.146565 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:45.294241 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.294209 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:45.647818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.647782 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:45.789953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.789926 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" Oct 30 13:34:45.789953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.789945 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt" event={"ID":"f31c2a89-ef80-463f-8150-2cd905423bcd","Type":"ContainerDied","Data":"d595642fc6e87f88ee4cdd1616da32aba807bc05fc4f12997f6052ed7ad84e03"} Oct 30 13:34:45.790195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.790007 5693 scope.go:122] "RemoveContainer" containerID="e1328e58bdb21a2b42f2a2b99f31f8a867e7143b965ce4c65318ed201aa96ef3" Oct 30 13:34:45.841401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.841368 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt"] Oct 30 13:34:45.852823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:45.852790 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-143cf2-b6f64c4d4-5blqt"] Oct 30 13:34:46.255290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:46.255257 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" path="/var/lib/kubelet/pods/4f3a03f2-e04a-4c64-af75-902820c8ff2b/volumes" Oct 30 13:34:46.255859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:46.255843 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" path="/var/lib/kubelet/pods/af5f7f81-8d55-42c6-8ff6-718e868da9d6/volumes" Oct 30 13:34:46.256265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:46.256251 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31c2a89-ef80-463f-8150-2cd905423bcd" path="/var/lib/kubelet/pods/f31c2a89-ef80-463f-8150-2cd905423bcd/volumes" Oct 30 13:34:46.257013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:46.256997 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" path="/var/lib/kubelet/pods/f533284d-9658-4d62-9502-43b0d7e9ce9b/volumes" Oct 30 13:34:46.995930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:46.995892 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:48.742758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:48.742468 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:49.910926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:49.910895 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:50.509737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:50.509700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:52.061588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:52.061554 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:52.395506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:52.395428 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:52.547057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:52.547020 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:52.685362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:52.685258 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2"] Oct 30 13:34:52.686134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:52.685588 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" containerID="cri-o://ed5692a8c1241b47df9fdf2377b5ae25e2e840a43aebc0d583052c3a5e9fbe0b" gracePeriod=30 Oct 30 13:34:52.755321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:52.755285 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:53.055978 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:53.055942 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:34:53.167827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.167786 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7680-0b49e0"] Oct 30 13:34:53.168495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168479 5693 cpu_manager.go:414] "containerMap: removing container" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168495 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168517 5693 cpu_manager.go:414] "containerMap: removing container" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168522 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168533 5693 cpu_manager.go:414] "containerMap: removing container" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168538 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168549 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" Oct 30 13:34:53.168564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168553 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168570 5693 cpu_manager.go:414] "containerMap: removing container" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168575 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168585 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168590 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168609 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f31c2a89-ef80-463f-8150-2cd905423bcd" containerName="maas-api" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168617 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31c2a89-ef80-463f-8150-2cd905423bcd" containerName="maas-api" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168758 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a4c160-ebaf-40a4-a5b5-f508d2aaba50" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168772 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5f7f81-8d55-42c6-8ff6-718e868da9d6" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168780 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="edafbadc-9d8c-4547-ba9d-b721c3b254a5" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168787 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3a03f2-e04a-4c64-af75-902820c8ff2b" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168796 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="239559a4-f5c0-4b59-85d3-3cd052a548de" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168802 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f533284d-9658-4d62-9502-43b0d7e9ce9b" containerName="istio-proxy" Oct 30 13:34:53.168804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168808 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31c2a89-ef80-463f-8150-2cd905423bcd" containerName="maas-api" Oct 30 13:34:53.169182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.168820 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a1b04f-20d2-4b26-961f-05bf8646844b" containerName="istio-proxy" Oct 30 13:34:53.174408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.174387 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:34:53.187956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.187928 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7680-0b49e0"] Oct 30 13:34:53.188570 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.188539 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4bkkd\"" Oct 30 13:34:53.322910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.322829 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm\") pod \"test-curl-7680-0b49e0\" (UID: \"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7\") " pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:34:53.424678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.424638 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm\") pod \"test-curl-7680-0b49e0\" (UID: \"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7\") " pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:34:53.444070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.444044 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm\") pod \"test-curl-7680-0b49e0\" (UID: \"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7\") " pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:34:53.487977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.487945 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:34:53.816674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.816641 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7680-0b49e0"] Oct 30 13:34:53.827843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.827816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:53.893747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:53.893720 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7680-0b49e0" event={"ID":"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7","Type":"ContainerStarted","Data":"eeabff8742a4159c6a9434b53bb6beafcf0e40341bf8748ac45e7aa9fe22251f"} Oct 30 13:34:54.535304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:54.535270 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq"] Oct 30 13:34:54.535834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:54.535475 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" containerID="cri-o://b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8" gracePeriod=30 Oct 30 13:34:54.903265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:54.903178 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7680-0b49e0" event={"ID":"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7","Type":"ContainerStarted","Data":"b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed"} Oct 30 13:34:54.926393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:54.926364 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:54.941420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:54.941374 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7680-0b49e0" podStartSLOduration=1.9413608839999998 podStartE2EDuration="1.941360884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:54.939294932 +0000 UTC m=+1870.805913107" watchObservedRunningTime="2026-10-30 13:34:54.941360884 +0000 UTC m=+1870.807979057" Oct 30 13:34:55.421209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.421174 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:55.753291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.753260 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:55.937990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.937943 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7680-0b49e0"] Oct 30 13:34:55.946722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.946687 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7680-0b49e0"] Oct 30 13:34:55.952628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.952265 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-3880ca"] Oct 30 13:34:55.962446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.952959 5693 cpu_manager.go:414] "containerMap: removing container" podUID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" containerName="test-curl-7680-0b49e0" Oct 30 13:34:55.962446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.952977 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" containerName="test-curl-7680-0b49e0" Oct 30 13:34:55.962446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.953125 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" containerName="test-curl-7680-0b49e0" Oct 30 13:34:55.962446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.956843 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:34:55.962997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:55.962968 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-3880ca"] Oct 30 13:34:56.089660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.089571 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd47p\" (UniqueName: \"kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p\") pod \"test-curl-7686-3880ca\" (UID: \"f90fd263-0b31-46be-af5e-11ca0acd2171\") " pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:34:56.191025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.190983 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd47p\" (UniqueName: \"kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p\") pod \"test-curl-7686-3880ca\" (UID: \"f90fd263-0b31-46be-af5e-11ca0acd2171\") " pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:34:56.209790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.209757 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd47p\" (UniqueName: \"kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p\") pod \"test-curl-7686-3880ca\" (UID: \"f90fd263-0b31-46be-af5e-11ca0acd2171\") " pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:34:56.281874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.281840 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:34:56.514820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.514787 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:56.591741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.591716 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-3880ca"] Oct 30 13:34:56.594063 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:34:56.594036 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf90fd263_0b31_46be_af5e_11ca0acd2171.slice/crio-123573536c2a3e67738c89b6c147020f42e5bcf6d87c107b1afd921ee4427d00 WatchSource:0}: Error finding container 123573536c2a3e67738c89b6c147020f42e5bcf6d87c107b1afd921ee4427d00: Status 404 returned error can't find the container with id 123573536c2a3e67738c89b6c147020f42e5bcf6d87c107b1afd921ee4427d00 Oct 30 13:34:56.901295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.901219 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58"] Oct 30 13:34:56.907003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.906971 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:56.927667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.927642 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58"] Oct 30 13:34:56.933104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.933058 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3f24fd-openshift-default-dockercfg-skptz\"" Oct 30 13:34:56.947969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.947929 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-3880ca" event={"ID":"f90fd263-0b31-46be-af5e-11ca0acd2171","Type":"ContainerStarted","Data":"296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6"} Oct 30 13:34:56.948113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.947982 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-3880ca" event={"ID":"f90fd263-0b31-46be-af5e-11ca0acd2171","Type":"ContainerStarted","Data":"123573536c2a3e67738c89b6c147020f42e5bcf6d87c107b1afd921ee4427d00"} Oct 30 13:34:56.948113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.948092 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7680-0b49e0" podUID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" containerName="test-curl-7680-0b49e0" containerID="cri-o://b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed" gracePeriod=2 Oct 30 13:34:56.988820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.988771 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-3880ca" podStartSLOduration=1.988758842 podStartE2EDuration="1.988758842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:56.984701333 +0000 UTC m=+1872.851319518" watchObservedRunningTime="2026-10-30 13:34:56.988758842 +0000 UTC m=+1872.855377014" Oct 30 13:34:56.990565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:56.990535 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:57.002660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.002334 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.002660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.002516 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.002660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.002637 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.003021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.002707 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.003021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.002759 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.003021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.002822 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.003200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.003081 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.003261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.003215 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxc6r\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.003261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.003248 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.106575 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.106674 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxc6r\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.106709 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.106791 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.106875 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.106987 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.107062 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.107117 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.107624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.107200 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.108265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.108013 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.113627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.108494 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.113627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.108749 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.113627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.109771 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.113627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.110021 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.136623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.132813 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:57.136623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.133740 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.145648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.139918 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.145648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.140454 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.145648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.141022 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxc6r\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r\") pod \"e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.239617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.237535 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:57.370633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.365574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:57.430565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.430469 5693 patch_prober.go:29] interesting pod/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:34:57.430565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.430519 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:34:57.663543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.663458 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58"] Oct 30 13:34:57.667760 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:34:57.667730 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6770ae55_0287_4dc9_a718_01f0e2136e8c.slice/crio-e5103028c1a1d60364c8c789908c2fb446c40574bc92d3a742f5cb8ffb124b51 WatchSource:0}: Error finding container e5103028c1a1d60364c8c789908c2fb446c40574bc92d3a742f5cb8ffb124b51: Status 404 returned error can't find the container with id e5103028c1a1d60364c8c789908c2fb446c40574bc92d3a742f5cb8ffb124b51 Oct 30 13:34:57.670954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.670921 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:34:57.671031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.670993 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:34:57.671031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.671024 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:34:57.962117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.960355 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" event={"ID":"6770ae55-0287-4dc9-a718-01f0e2136e8c","Type":"ContainerStarted","Data":"2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244"} Oct 30 13:34:57.962117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.960417 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" event={"ID":"6770ae55-0287-4dc9-a718-01f0e2136e8c","Type":"ContainerStarted","Data":"e5103028c1a1d60364c8c789908c2fb446c40574bc92d3a742f5cb8ffb124b51"} Oct 30 13:34:57.962777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.962750 5693 generic.go:356] "Generic (PLEG): container finished" podID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerID="ed5692a8c1241b47df9fdf2377b5ae25e2e840a43aebc0d583052c3a5e9fbe0b" exitCode=0 Oct 30 13:34:57.962884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.962821 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" event={"ID":"91110e47-3f71-4d75-bccd-bf1c5455b5e3","Type":"ContainerDied","Data":"ed5692a8c1241b47df9fdf2377b5ae25e2e840a43aebc0d583052c3a5e9fbe0b"} Oct 30 13:34:57.962884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.962878 5693 scope.go:122] "RemoveContainer" containerID="5647050206aedbdb0ec64ad2a7613cfec15a3ea5efb7ae93f36ebebca1834da6" Oct 30 13:34:57.969321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:57.969298 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:34:58.002698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.002652 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podStartSLOduration=2.002639474 podStartE2EDuration="2.002639474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:58.000274506 +0000 UTC m=+1873.866892700" watchObservedRunningTime="2026-10-30 13:34:58.002639474 +0000 UTC m=+1873.869257643" Oct 30 13:34:58.061589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.061553 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-3880ca"] Oct 30 13:34:58.068895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.068859 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-3880ca"] Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145204 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145297 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145372 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145431 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145798 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145844 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.145897 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.146069 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.156734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.146156 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm\") pod \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\" (UID: \"91110e47-3f71-4d75-bccd-bf1c5455b5e3\") " Oct 30 13:34:58.171806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.171757 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:58.172086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.172066 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:58.203666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.190292 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:58.207630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.204787 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:58.207630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.207222 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:58.226689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.226539 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-82dc56"] Oct 30 13:34:58.228021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.227978 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" containerName="test-curl-7686-3880ca" Oct 30 13:34:58.231657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.231636 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" containerName="test-curl-7686-3880ca" Oct 30 13:34:58.231790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.231774 5693 cpu_manager.go:414] "containerMap: removing container" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" Oct 30 13:34:58.231879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.231867 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" Oct 30 13:34:58.232009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.231997 5693 cpu_manager.go:414] "containerMap: removing container" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" Oct 30 13:34:58.232091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.232079 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" Oct 30 13:34:58.232390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.232372 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" Oct 30 13:34:58.232533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.232519 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" containerName="test-curl-7686-3880ca" Oct 30 13:34:58.244621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.238462 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:34:58.244621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.243180 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:58.251980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.249477 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.251980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.249506 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.251980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.249521 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.251980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.249536 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.251980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.249549 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.253645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.253617 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:34:58.253748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.253669 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:34:58.269397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.267090 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "kube-api-access-dvzmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:58.269397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.267643 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:58.269397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.267785 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:58.270802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.270753 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token" pod "91110e47-3f71-4d75-bccd-bf1c5455b5e3" (UID: "91110e47-3f71-4d75-bccd-bf1c5455b5e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:58.291844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.291820 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-82dc56"] Oct 30 13:34:58.350683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.350649 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269fn\" (UniqueName: \"kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn\") pod \"test-curl-7686-82dc56\" (UID: \"14a0432d-97fd-4258-8519-ac9b918662e3\") " pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:34:58.350940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.350925 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.351004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.350943 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzmm\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-kube-api-access-dvzmm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.351004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.350953 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.351004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.350961 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91110e47-3f71-4d75-bccd-bf1c5455b5e3-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:58.452006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.451977 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269fn\" (UniqueName: \"kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn\") pod \"test-curl-7686-82dc56\" (UID: \"14a0432d-97fd-4258-8519-ac9b918662e3\") " pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:34:58.471471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.471439 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269fn\" (UniqueName: \"kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn\") pod \"test-curl-7686-82dc56\" (UID: \"14a0432d-97fd-4258-8519-ac9b918662e3\") " pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:34:58.602179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.602101 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:34:58.904244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.904214 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-82dc56"] Oct 30 13:34:58.906819 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:34:58.906789 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14a0432d_97fd_4258_8519_ac9b918662e3.slice/crio-8ada9d0e7c8683110578039a0dd5b437195a2cb4b642b1bec9b54e3d0b9d8893 WatchSource:0}: Error finding container 8ada9d0e7c8683110578039a0dd5b437195a2cb4b642b1bec9b54e3d0b9d8893: Status 404 returned error can't find the container with id 8ada9d0e7c8683110578039a0dd5b437195a2cb4b642b1bec9b54e3d0b9d8893 Oct 30 13:34:58.928576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:58.928534 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:59.000513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.000479 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" Oct 30 13:34:59.000951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.000783 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2" event={"ID":"91110e47-3f71-4d75-bccd-bf1c5455b5e3","Type":"ContainerDied","Data":"b2944dcc566ce3c84465e1c4355c23c33af5002d0f7358bea8493297a77fb115"} Oct 30 13:34:59.000951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.000849 5693 scope.go:122] "RemoveContainer" containerID="ed5692a8c1241b47df9fdf2377b5ae25e2e840a43aebc0d583052c3a5e9fbe0b" Oct 30 13:34:59.003067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.003038 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-3880ca" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" containerName="test-curl-7686-3880ca" containerID="cri-o://296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6" gracePeriod=2 Oct 30 13:34:59.003320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.003297 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-82dc56" event={"ID":"14a0432d-97fd-4258-8519-ac9b918662e3","Type":"ContainerStarted","Data":"8ada9d0e7c8683110578039a0dd5b437195a2cb4b642b1bec9b54e3d0b9d8893"} Oct 30 13:34:59.104539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.104505 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2"] Oct 30 13:34:59.109618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.109571 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-143cf2-openshift-default-7495cdfc89-t8nv2"] Oct 30 13:34:59.235228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.235205 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:34:59.243819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.243789 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:34:59.365540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.365509 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm\") pod \"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7\" (UID: \"98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7\") " Oct 30 13:34:59.368328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.368294 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm" pod "98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" (UID: "98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7"). InnerVolumeSpecName "kube-api-access-rrjnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:59.468272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.468186 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7-kube-api-access-rrjnm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:34:59.597612 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:34:59.597568 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d2f787_e433_41a5_ac92_43b3db7654a5.slice/crio-b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:34:59.821622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.821568 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:34:59.975104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.975049 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.975756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.975705 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.976145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.976122 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.976537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.976514 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.976726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.976704 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.979894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.979845 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.980351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.980328 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.980576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.980558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.980910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.980892 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs\") pod \"32d2f787-e433-41a5-ac92-43b3db7654a5\" (UID: \"32d2f787-e433-41a5-ac92-43b3db7654a5\") " Oct 30 13:34:59.982767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.982698 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.983991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.983964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.984549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.984526 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.984706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.984678 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:59.984928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.984909 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.986263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.986238 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:59.986553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.986531 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.986843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.986820 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:34:59.986953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:34:59.986915 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk" pod "32d2f787-e433-41a5-ac92-43b3db7654a5" (UID: "32d2f787-e433-41a5-ac92-43b3db7654a5"). InnerVolumeSpecName "kube-api-access-h7jvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:00.012073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.012042 5693 generic.go:356] "Generic (PLEG): container finished" podID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" containerID="b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed" exitCode=137 Oct 30 13:35:00.012502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.012084 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:35:00.012502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.012135 5693 scope.go:122] "RemoveContainer" containerID="b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed" Oct 30 13:35:00.013669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.013642 5693 generic.go:356] "Generic (PLEG): container finished" podID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerID="b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8" exitCode=0 Oct 30 13:35:00.013789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.013706 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" event={"ID":"32d2f787-e433-41a5-ac92-43b3db7654a5","Type":"ContainerDied","Data":"b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8"} Oct 30 13:35:00.013789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.013654 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" Oct 30 13:35:00.013789 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.013753 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq" event={"ID":"32d2f787-e433-41a5-ac92-43b3db7654a5","Type":"ContainerDied","Data":"1dc8a41a3170e6f70985149e60769234cd698332bc4344ad2b0d35eac5004dc6"} Oct 30 13:35:00.016405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.016379 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-82dc56" event={"ID":"14a0432d-97fd-4258-8519-ac9b918662e3","Type":"ContainerStarted","Data":"eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43"} Oct 30 13:35:00.016706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.016673 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:35:00.017665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.017646 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:35:00.023493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.023246 5693 scope.go:122] "RemoveContainer" containerID="b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed" Oct 30 13:35:00.023679 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:00.023576 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed\": container with ID starting with b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed not found: ID does not exist" containerID="b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed" Oct 30 13:35:00.023782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.023721 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed"} err="failed to get container status \"b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed\": rpc error: code = NotFound desc = could not find container \"b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed\": container with ID starting with b95f3e191c869a60b24465d16e486ad57b6057955d7ea08457bf1e3be938bfed not found: ID does not exist" Oct 30 13:35:00.023782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.023745 5693 scope.go:122] "RemoveContainer" containerID="b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8" Oct 30 13:35:00.035489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.035470 5693 scope.go:122] "RemoveContainer" containerID="b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8" Oct 30 13:35:00.035849 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:00.035819 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8\": container with ID starting with b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8 not found: ID does not exist" containerID="b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8" Oct 30 13:35:00.035971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.035857 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8"} err="failed to get container status \"b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8\": rpc error: code = NotFound desc = could not find container \"b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8\": container with ID starting with b4d2f6ff30b003a8ab181f7a9e8e5138ae7e1eb297ffaae0ccd86fb708a68ad8 not found: ID does not exist" Oct 30 13:35:00.067462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.067389 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq"] Oct 30 13:35:00.073330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.073306 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-lkxnq"] Oct 30 13:35:00.083292 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:00.083261 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-0b49e0\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" pod="opendatahub/test-curl-7680-0b49e0" Oct 30 13:35:00.083981 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.083958 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.083981 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.083979 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.083991 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.083998 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.084006 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.084014 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.084024 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d2f787-e433-41a5-ac92-43b3db7654a5-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.084033 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7jvk\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-kube-api-access-h7jvk\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.084160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.084041 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d2f787-e433-41a5-ac92-43b3db7654a5-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.122312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.122261 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-82dc56" podStartSLOduration=2.122250342 podStartE2EDuration="2.122250342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:00.121446753 +0000 UTC m=+1875.988064929" watchObservedRunningTime="2026-10-30 13:35:00.122250342 +0000 UTC m=+1875.988868514" Oct 30 13:35:00.256644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.256583 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" path="/var/lib/kubelet/pods/32d2f787-e433-41a5-ac92-43b3db7654a5/volumes" Oct 30 13:35:00.257149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.257136 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" path="/var/lib/kubelet/pods/91110e47-3f71-4d75-bccd-bf1c5455b5e3/volumes" Oct 30 13:35:00.257492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.257480 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7" path="/var/lib/kubelet/pods/98c36f95-ee24-4ea8-834d-7f4c7d8a9bf7/volumes" Oct 30 13:35:00.477838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.477801 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:00.650550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.650519 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:00.874676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.872992 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:00.988085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.988053 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-82dc56"] Oct 30 13:35:00.992703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:00.992673 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-82dc56"] Oct 30 13:35:01.168153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.168121 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-7085ec"] Oct 30 13:35:01.169213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169185 5693 cpu_manager.go:414] "containerMap: removing container" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" containerName="test-curl-7686-82dc56" Oct 30 13:35:01.169213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169212 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" containerName="test-curl-7686-82dc56" Oct 30 13:35:01.169424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169283 5693 cpu_manager.go:414] "containerMap: removing container" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" Oct 30 13:35:01.169424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169291 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" Oct 30 13:35:01.169541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169440 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="91110e47-3f71-4d75-bccd-bf1c5455b5e3" containerName="istio-proxy" Oct 30 13:35:01.169541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169461 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d2f787-e433-41a5-ac92-43b3db7654a5" containerName="istio-proxy" Oct 30 13:35:01.169541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.169477 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" containerName="test-curl-7686-82dc56" Oct 30 13:35:01.173858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.173838 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:01.190087 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.190060 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-7085ec"] Oct 30 13:35:01.260889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.260864 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:35:01.314372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.314339 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7vd\" (UniqueName: \"kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd\") pod \"test-curl-7686-7085ec\" (UID: \"9ce62b68-26a6-44fe-8f80-22bfb92127f3\") " pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:01.415463 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.415372 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p\" (UniqueName: \"kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p\") pod \"f90fd263-0b31-46be-af5e-11ca0acd2171\" (UID: \"f90fd263-0b31-46be-af5e-11ca0acd2171\") " Oct 30 13:35:01.416344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.416317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7vd\" (UniqueName: \"kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd\") pod \"test-curl-7686-7085ec\" (UID: \"9ce62b68-26a6-44fe-8f80-22bfb92127f3\") " pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:01.417878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.417850 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p" pod "f90fd263-0b31-46be-af5e-11ca0acd2171" (UID: "f90fd263-0b31-46be-af5e-11ca0acd2171"). InnerVolumeSpecName "kube-api-access-gd47p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:01.435991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.435968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7vd\" (UniqueName: \"kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd\") pod \"test-curl-7686-7085ec\" (UID: \"9ce62b68-26a6-44fe-8f80-22bfb92127f3\") " pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:01.489656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.489630 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:01.517926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.517894 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd47p\" (UniqueName: \"kubernetes.io/projected/f90fd263-0b31-46be-af5e-11ca0acd2171-kube-api-access-gd47p\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.792643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:01.792613 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-7085ec"] Oct 30 13:35:01.794798 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:01.794768 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ce62b68_26a6_44fe_8f80_22bfb92127f3.slice/crio-1fae08a6a2aa30b39d690dc987f31d6201f77e251c1a97759916a5ac07bdd9f8 WatchSource:0}: Error finding container 1fae08a6a2aa30b39d690dc987f31d6201f77e251c1a97759916a5ac07bdd9f8: Status 404 returned error can't find the container with id 1fae08a6a2aa30b39d690dc987f31d6201f77e251c1a97759916a5ac07bdd9f8 Oct 30 13:35:02.036354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.036313 5693 generic.go:356] "Generic (PLEG): container finished" podID="f90fd263-0b31-46be-af5e-11ca0acd2171" containerID="296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6" exitCode=137 Oct 30 13:35:02.036513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.036360 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:35:02.036513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.036421 5693 scope.go:122] "RemoveContainer" containerID="296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6" Oct 30 13:35:02.038423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.038394 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-7085ec" event={"ID":"9ce62b68-26a6-44fe-8f80-22bfb92127f3","Type":"ContainerStarted","Data":"570b4a6aec31ea766c75f8adb6d0c731a8ff2cded8520dd4e753f0e4cc4ca25b"} Oct 30 13:35:02.038577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.038442 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-7085ec" event={"ID":"9ce62b68-26a6-44fe-8f80-22bfb92127f3","Type":"ContainerStarted","Data":"1fae08a6a2aa30b39d690dc987f31d6201f77e251c1a97759916a5ac07bdd9f8"} Oct 30 13:35:02.038869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.038845 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-82dc56" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" containerName="test-curl-7686-82dc56" containerID="cri-o://eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43" gracePeriod=2 Oct 30 13:35:02.048057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.048034 5693 scope.go:122] "RemoveContainer" containerID="296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6" Oct 30 13:35:02.048423 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.048394 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6\": container with ID starting with 296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6 not found: ID does not exist" containerID="296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6" Oct 30 13:35:02.048503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.048425 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6"} err="failed to get container status \"296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6\": rpc error: code = NotFound desc = could not find container \"296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6\": container with ID starting with 296fdf90c10deaf59b29bd41dedfd468bca55bc7520d85a34b12a9adac2d48f6 not found: ID does not exist" Oct 30 13:35:02.081881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.081827 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-7085ec" podStartSLOduration=1.081811041 podStartE2EDuration="1.081811041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:02.079301854 +0000 UTC m=+1877.945920030" watchObservedRunningTime="2026-10-30 13:35:02.081811041 +0000 UTC m=+1877.948429214" Oct 30 13:35:02.092362 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.092304 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:02.099460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.099402 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:02.106211 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.106187 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3880ca\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:35:02.163939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.163901 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll"] Oct 30 13:35:02.169764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.169736 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.185504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.185475 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll"] Oct 30 13:35:02.186790 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.186758 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3880ca\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:35:02.189732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.189704 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-3f24fd-dockercfg-mcdg4\"" Oct 30 13:35:02.193580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.193557 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-3f24fd\"" Oct 30 13:35:02.193713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.193580 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg"] Oct 30 13:35:02.198397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.198375 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.224844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.224810 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg"] Oct 30 13:35:02.226113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.226085 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.226239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.226133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.226301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.226267 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmsqw\" (UniqueName: \"kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw\") pod \"payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.226441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.226425 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5vt\" (UniqueName: \"kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt\") pod \"payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.240084 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.240051 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.341159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.349626 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsqw\" (UniqueName: \"kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw\") pod \"payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.374051 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5vt\" (UniqueName: \"kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt\") pod \"payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.374143 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.374235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.375273 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.376650 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.394674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.384483 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" path="/var/lib/kubelet/pods/f90fd263-0b31-46be-af5e-11ca0acd2171/volumes" Oct 30 13:35:02.460177 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.459521 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:02.478638 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:02.474793 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3880ca\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="f90fd263-0b31-46be-af5e-11ca0acd2171" pod="opendatahub/test-curl-7686-3880ca" Oct 30 13:35:02.478638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.478288 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5vt\" (UniqueName: \"kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt\") pod \"payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.480272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.480239 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsqw\" (UniqueName: \"kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw\") pod \"payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.489406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.489375 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:02.540278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.540021 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:02.824025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.823991 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll"] Oct 30 13:35:02.827811 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:02.827777 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a41edd2_d1ed_4bc9_95bb_8491eab9dd12.slice/crio-85aa47dcc16c48273cf7a3396e3475857358cd8417d56fa4385862e731c0851b WatchSource:0}: Error finding container 85aa47dcc16c48273cf7a3396e3475857358cd8417d56fa4385862e731c0851b: Status 404 returned error can't find the container with id 85aa47dcc16c48273cf7a3396e3475857358cd8417d56fa4385862e731c0851b Oct 30 13:35:02.893913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.881032 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:02.893913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:02.886558 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg"] Oct 30 13:35:02.904284 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:02.904249 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895e8e9a_af7b_4d49_9132_5c494886f00f.slice/crio-3f373cd81bc89afb0dce2464638f72abe9948fd303568714dde47384f151819d WatchSource:0}: Error finding container 3f373cd81bc89afb0dce2464638f72abe9948fd303568714dde47384f151819d: Status 404 returned error can't find the container with id 3f373cd81bc89afb0dce2464638f72abe9948fd303568714dde47384f151819d Oct 30 13:35:03.048513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.048479 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" event={"ID":"895e8e9a-af7b-4d49-9132-5c494886f00f","Type":"ContainerStarted","Data":"92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f"} Oct 30 13:35:03.048700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.048537 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" event={"ID":"895e8e9a-af7b-4d49-9132-5c494886f00f","Type":"ContainerStarted","Data":"3f373cd81bc89afb0dce2464638f72abe9948fd303568714dde47384f151819d"} Oct 30 13:35:03.048766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.048710 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:03.050160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.050130 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" event={"ID":"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12","Type":"ContainerStarted","Data":"6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7"} Oct 30 13:35:03.050286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.050182 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" event={"ID":"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12","Type":"ContainerStarted","Data":"85aa47dcc16c48273cf7a3396e3475857358cd8417d56fa4385862e731c0851b"} Oct 30 13:35:03.050286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.050212 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:03.091692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.091582 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" podStartSLOduration=1.091567798 podStartE2EDuration="1.091567798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:03.087870812 +0000 UTC m=+1878.954488985" watchObservedRunningTime="2026-10-30 13:35:03.091567798 +0000 UTC m=+1878.958185967" Oct 30 13:35:03.126670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.126585 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" podStartSLOduration=1.1265717450000001 podStartE2EDuration="1.126571745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:03.125979124 +0000 UTC m=+1878.992597298" watchObservedRunningTime="2026-10-30 13:35:03.126571745 +0000 UTC m=+1878.993189917" Oct 30 13:35:03.195037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.194961 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-7085ec"] Oct 30 13:35:03.199918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.199877 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-7085ec"] Oct 30 13:35:03.340487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.340452 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-310345"] Oct 30 13:35:03.341381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.341358 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" containerName="test-curl-7686-7085ec" Oct 30 13:35:03.341381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.341378 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" containerName="test-curl-7686-7085ec" Oct 30 13:35:03.341521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.341514 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" containerName="test-curl-7686-7085ec" Oct 30 13:35:03.345391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.345338 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:03.363069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.363041 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-310345"] Oct 30 13:35:03.406637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.406588 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zd4k\" (UniqueName: \"kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k\") pod \"test-curl-7686-310345\" (UID: \"7bd12f40-2d9e-4fd8-b611-18107c7381b9\") " pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:03.508191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.508136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zd4k\" (UniqueName: \"kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k\") pod \"test-curl-7686-310345\" (UID: \"7bd12f40-2d9e-4fd8-b611-18107c7381b9\") " pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:03.527143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.527114 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zd4k\" (UniqueName: \"kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k\") pod \"test-curl-7686-310345\" (UID: \"7bd12f40-2d9e-4fd8-b611-18107c7381b9\") " pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:03.659410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.659320 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:03.963061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:03.963032 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-310345"] Oct 30 13:35:03.965470 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:03.965444 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd12f40_2d9e_4fd8_b611_18107c7381b9.slice/crio-9b2b68cfa7b5faa7703d879ff66c4183d486ed4e7d3103e71ddd404a324265a9 WatchSource:0}: Error finding container 9b2b68cfa7b5faa7703d879ff66c4183d486ed4e7d3103e71ddd404a324265a9: Status 404 returned error can't find the container with id 9b2b68cfa7b5faa7703d879ff66c4183d486ed4e7d3103e71ddd404a324265a9 Oct 30 13:35:04.033856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.033822 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr"] Oct 30 13:35:04.036933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.036906 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d83bf2-xj8mr"] Oct 30 13:35:04.058140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.058104 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-310345" event={"ID":"7bd12f40-2d9e-4fd8-b611-18107c7381b9","Type":"ContainerStarted","Data":"464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385"} Oct 30 13:35:04.058233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.058157 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-310345" event={"ID":"7bd12f40-2d9e-4fd8-b611-18107c7381b9","Type":"ContainerStarted","Data":"9b2b68cfa7b5faa7703d879ff66c4183d486ed4e7d3103e71ddd404a324265a9"} Oct 30 13:35:04.058303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.058219 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-7085ec" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" containerName="test-curl-7686-7085ec" containerID="cri-o://570b4a6aec31ea766c75f8adb6d0c731a8ff2cded8520dd4e753f0e4cc4ca25b" gracePeriod=2 Oct 30 13:35:04.095280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.095231 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-310345" podStartSLOduration=1.095214414 podStartE2EDuration="1.095214414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:04.094754177 +0000 UTC m=+1879.961372349" watchObservedRunningTime="2026-10-30 13:35:04.095214414 +0000 UTC m=+1879.961832588" Oct 30 13:35:04.108675 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:04.108638 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-7085ec\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:04.121738 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:04.121709 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-7085ec\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:04.259800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.259771 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b13c109-e844-4563-9824-6ddcf3728566" path="/var/lib/kubelet/pods/7b13c109-e844-4563-9824-6ddcf3728566/volumes" Oct 30 13:35:04.265711 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:04.265684 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-7085ec\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:04.279997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.279977 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:04.294390 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:04.294354 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:04.320068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.320040 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn\" (UniqueName: \"kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn\") pod \"14a0432d-97fd-4258-8519-ac9b918662e3\" (UID: \"14a0432d-97fd-4258-8519-ac9b918662e3\") " Oct 30 13:35:04.322252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.322231 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn" pod "14a0432d-97fd-4258-8519-ac9b918662e3" (UID: "14a0432d-97fd-4258-8519-ac9b918662e3"). InnerVolumeSpecName "kube-api-access-269fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:04.422332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.422300 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269fn\" (UniqueName: \"kubernetes.io/projected/14a0432d-97fd-4258-8519-ac9b918662e3-kube-api-access-269fn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:04.534848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.534616 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:04.926229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.926152 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-310345"] Oct 30 13:35:04.930424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:04.930391 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-310345"] Oct 30 13:35:05.067180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.067142 5693 generic.go:356] "Generic (PLEG): container finished" podID="14a0432d-97fd-4258-8519-ac9b918662e3" containerID="eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43" exitCode=137 Oct 30 13:35:05.067180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.067183 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:05.067417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.067237 5693 scope.go:122] "RemoveContainer" containerID="eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43" Oct 30 13:35:05.078149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.078121 5693 scope.go:122] "RemoveContainer" containerID="eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43" Oct 30 13:35:05.078522 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:05.078496 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43\": container with ID starting with eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43 not found: ID does not exist" containerID="eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43" Oct 30 13:35:05.078575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.078532 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43"} err="failed to get container status \"eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43\": rpc error: code = NotFound desc = could not find container \"eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43\": container with ID starting with eaf8f9e38ce1460a2984cad2257b7d2261a44e29fa78c4383ff59d336ea02b43 not found: ID does not exist" Oct 30 13:35:05.080367 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:05.080339 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:05.080937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.080916 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-182dd0"] Oct 30 13:35:05.081673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.081657 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" containerName="test-curl-7686-310345" Oct 30 13:35:05.081718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.081674 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" containerName="test-curl-7686-310345" Oct 30 13:35:05.081794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.081786 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" containerName="test-curl-7686-310345" Oct 30 13:35:05.086810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.086787 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:05.093404 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:05.093372 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:05.102493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.102457 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-182dd0"] Oct 30 13:35:05.106585 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:05.106526 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:05.143426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.143373 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4g5v\" (UniqueName: \"kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v\") pod \"test-curl-7686-182dd0\" (UID: \"aec4ff34-8065-4644-a311-3e5972c63fc1\") " pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:05.244429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.244397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4g5v\" (UniqueName: \"kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v\") pod \"test-curl-7686-182dd0\" (UID: \"aec4ff34-8065-4644-a311-3e5972c63fc1\") " pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:05.264315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.264283 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4g5v\" (UniqueName: \"kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v\") pod \"test-curl-7686-182dd0\" (UID: \"aec4ff34-8065-4644-a311-3e5972c63fc1\") " pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:05.404781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.404747 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:05.707409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:05.707377 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-182dd0"] Oct 30 13:35:05.710469 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:05.710442 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaec4ff34_8065_4644_a311_3e5972c63fc1.slice/crio-b7dd9ea92e84daaab690ce19d64ef0eab032f1eeb713f26377a9f5594ddc578c WatchSource:0}: Error finding container b7dd9ea92e84daaab690ce19d64ef0eab032f1eeb713f26377a9f5594ddc578c: Status 404 returned error can't find the container with id b7dd9ea92e84daaab690ce19d64ef0eab032f1eeb713f26377a9f5594ddc578c Oct 30 13:35:06.077881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.077856 5693 generic.go:356] "Generic (PLEG): container finished" podID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" containerID="570b4a6aec31ea766c75f8adb6d0c731a8ff2cded8520dd4e753f0e4cc4ca25b" exitCode=137 Oct 30 13:35:06.079552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.079524 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-182dd0" event={"ID":"aec4ff34-8065-4644-a311-3e5972c63fc1","Type":"ContainerStarted","Data":"22d804811392040ce7fb14f0ce90175523d5f1eb99d8d72648e0fef16d0cb7c3"} Oct 30 13:35:06.079696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.079564 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-182dd0" event={"ID":"aec4ff34-8065-4644-a311-3e5972c63fc1","Type":"ContainerStarted","Data":"b7dd9ea92e84daaab690ce19d64ef0eab032f1eeb713f26377a9f5594ddc578c"} Oct 30 13:35:06.080827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.080806 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-310345" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" containerName="test-curl-7686-310345" containerID="cri-o://464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385" gracePeriod=2 Oct 30 13:35:06.136153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.136077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:06.161623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.153123 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-182dd0" podStartSLOduration=1.153097147 podStartE2EDuration="1.153097147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:06.135090202 +0000 UTC m=+1882.001708387" watchObservedRunningTime="2026-10-30 13:35:06.153097147 +0000 UTC m=+1882.019715324" Oct 30 13:35:06.161623 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:06.155109 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:06.175955 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:06.174815 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-310345\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:06.187844 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:06.187807 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-82dc56\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" pod="opendatahub/test-curl-7686-82dc56" Oct 30 13:35:06.257300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.257262 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a0432d-97fd-4258-8519-ac9b918662e3" path="/var/lib/kubelet/pods/14a0432d-97fd-4258-8519-ac9b918662e3/volumes" Oct 30 13:35:06.262952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.262927 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:06.358825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.358739 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd\" (UniqueName: \"kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd\") pod \"9ce62b68-26a6-44fe-8f80-22bfb92127f3\" (UID: \"9ce62b68-26a6-44fe-8f80-22bfb92127f3\") " Oct 30 13:35:06.361029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.361002 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd" pod "9ce62b68-26a6-44fe-8f80-22bfb92127f3" (UID: "9ce62b68-26a6-44fe-8f80-22bfb92127f3"). InnerVolumeSpecName "kube-api-access-4m7vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:06.461341 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.461307 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7vd\" (UniqueName: \"kubernetes.io/projected/9ce62b68-26a6-44fe-8f80-22bfb92127f3-kube-api-access-4m7vd\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:06.978085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.978008 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-182dd0"] Oct 30 13:35:06.982208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:06.982177 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-182dd0"] Oct 30 13:35:07.089630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:07.089572 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:07.089630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:07.089632 5693 scope.go:122] "RemoveContainer" containerID="570b4a6aec31ea766c75f8adb6d0c731a8ff2cded8520dd4e753f0e4cc4ca25b" Oct 30 13:35:07.112098 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:07.112065 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-7085ec\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:07.684676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:07.684643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:08.099778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.099738 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-182dd0" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" containerName="test-curl-7686-182dd0" containerID="cri-o://22d804811392040ce7fb14f0ce90175523d5f1eb99d8d72648e0fef16d0cb7c3" gracePeriod=2 Oct 30 13:35:08.112780 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:08.112752 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-182dd0\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:08.125722 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:08.125687 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-7085ec\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" pod="opendatahub/test-curl-7686-7085ec" Oct 30 13:35:08.328367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.328324 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce62b68-26a6-44fe-8f80-22bfb92127f3" path="/var/lib/kubelet/pods/9ce62b68-26a6-44fe-8f80-22bfb92127f3/volumes" Oct 30 13:35:08.588340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.588303 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:08.612746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.612713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:08.721996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.718721 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k\" (UniqueName: \"kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k\") pod \"7bd12f40-2d9e-4fd8-b611-18107c7381b9\" (UID: \"7bd12f40-2d9e-4fd8-b611-18107c7381b9\") " Oct 30 13:35:08.740833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.739316 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k" pod "7bd12f40-2d9e-4fd8-b611-18107c7381b9" (UID: "7bd12f40-2d9e-4fd8-b611-18107c7381b9"). InnerVolumeSpecName "kube-api-access-6zd4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:08.828622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.821249 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zd4k\" (UniqueName: \"kubernetes.io/projected/7bd12f40-2d9e-4fd8-b611-18107c7381b9-kube-api-access-6zd4k\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:08.982506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.982467 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp"] Oct 30 13:35:08.983714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.983688 5693 cpu_manager.go:414] "containerMap: removing container" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" containerName="test-curl-7686-182dd0" Oct 30 13:35:08.983714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.983714 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" containerName="test-curl-7686-182dd0" Oct 30 13:35:08.983895 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:08.983880 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" containerName="test-curl-7686-182dd0" Oct 30 13:35:09.021436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.021400 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp"] Oct 30 13:35:09.021657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.021540 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.044902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.044300 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cad2c6-openshift-default-dockercfg-gdmvg\"" Oct 30 13:35:09.078047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.078014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:09.161632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.136609 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.161989 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162125 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162275 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162335 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162469 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162502 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqwp\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.162854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.162538 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.214624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.197550 5693 generic.go:356] "Generic (PLEG): container finished" podID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" containerID="464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385" exitCode=137 Oct 30 13:35:09.215272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.215246 5693 scope.go:122] "RemoveContainer" containerID="464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385" Oct 30 13:35:09.215914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.215802 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270490 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270665 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqwp\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270701 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270836 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270859 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.270939 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.271011 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.271059 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.271513 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.271803 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.272038 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.286408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.272975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.289436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.287568 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.289926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.289847 5693 scope.go:122] "RemoveContainer" containerID="464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385" Oct 30 13:35:09.294628 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:09.294137 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385\": container with ID starting with 464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385 not found: ID does not exist" containerID="464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385" Oct 30 13:35:09.294628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.294175 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385"} err="failed to get container status \"464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385\": rpc error: code = NotFound desc = could not find container \"464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385\": container with ID starting with 464ac81c65070d1fccc33d3473b9bd53774955013167994d32ce8e0cb36b5385 not found: ID does not exist" Oct 30 13:35:09.310784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.310580 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.311933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.311903 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqwp\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.330369 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:09.316297 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-310345\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:09.332316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.332270 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.337970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.337910 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token\") pod \"e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.347440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.347146 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:09.521707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.521572 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:09.751934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.751822 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp"] Oct 30 13:35:09.754396 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:09.754367 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0512a21c_cfd6_47ec_9c02_4e2079109cc2.slice/crio-3554027dc45e7c4d9b1a50203af833996b76acc5ccc05d861555d921ffa18c43 WatchSource:0}: Error finding container 3554027dc45e7c4d9b1a50203af833996b76acc5ccc05d861555d921ffa18c43: Status 404 returned error can't find the container with id 3554027dc45e7c4d9b1a50203af833996b76acc5ccc05d861555d921ffa18c43 Oct 30 13:35:09.757252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.757218 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:09.757323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.757289 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:09.757323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.757318 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:09.793358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:09.793242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:10.226535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.226498 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" event={"ID":"0512a21c-cfd6-47ec-9c02-4e2079109cc2","Type":"ContainerStarted","Data":"3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131"} Oct 30 13:35:10.226752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.226589 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" event={"ID":"0512a21c-cfd6-47ec-9c02-4e2079109cc2","Type":"ContainerStarted","Data":"3554027dc45e7c4d9b1a50203af833996b76acc5ccc05d861555d921ffa18c43"} Oct 30 13:35:10.228540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.228505 5693 generic.go:356] "Generic (PLEG): container finished" podID="aec4ff34-8065-4644-a311-3e5972c63fc1" containerID="22d804811392040ce7fb14f0ce90175523d5f1eb99d8d72648e0fef16d0cb7c3" exitCode=137 Oct 30 13:35:10.239675 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:10.239639 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-310345\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" pod="opendatahub/test-curl-7686-310345" Oct 30 13:35:10.269656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.269621 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd12f40-2d9e-4fd8-b611-18107c7381b9" path="/var/lib/kubelet/pods/7bd12f40-2d9e-4fd8-b611-18107c7381b9/volumes" Oct 30 13:35:10.289777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.289720 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podStartSLOduration=2.289704205 podStartE2EDuration="2.289704205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:10.282577645 +0000 UTC m=+1886.149195817" watchObservedRunningTime="2026-10-30 13:35:10.289704205 +0000 UTC m=+1886.156322379" Oct 30 13:35:10.339140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.339115 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:10.348723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.348704 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:10.354889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.354832 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:10.514161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.514132 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v\" (UniqueName: \"kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v\") pod \"aec4ff34-8065-4644-a311-3e5972c63fc1\" (UID: \"aec4ff34-8065-4644-a311-3e5972c63fc1\") " Oct 30 13:35:10.516474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.516446 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v" pod "aec4ff34-8065-4644-a311-3e5972c63fc1" (UID: "aec4ff34-8065-4644-a311-3e5972c63fc1"). InnerVolumeSpecName "kube-api-access-l4g5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:10.616275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:10.616204 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4g5v\" (UniqueName: \"kubernetes.io/projected/aec4ff34-8065-4644-a311-3e5972c63fc1-kube-api-access-l4g5v\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:11.240378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:11.240286 5693 scope.go:122] "RemoveContainer" containerID="22d804811392040ce7fb14f0ce90175523d5f1eb99d8d72648e0fef16d0cb7c3" Oct 30 13:35:11.240378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:11.240300 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:11.241619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:11.241581 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:11.242643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:11.242624 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:35:11.311178 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:11.311139 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-182dd0\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" pod="opendatahub/test-curl-7686-182dd0" Oct 30 13:35:11.511499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:11.511336 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:12.262442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:12.262399 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec4ff34-8065-4644-a311-3e5972c63fc1" path="/var/lib/kubelet/pods/aec4ff34-8065-4644-a311-3e5972c63fc1/volumes" Oct 30 13:35:12.738954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:12.738910 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:12.890286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:12.890255 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:13.508586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:13.508555 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:14.061332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.061303 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 13:35:14.061608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.061369 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 13:35:14.438809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.438738 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:14.500684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.500654 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk"] Oct 30 13:35:14.506790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.506769 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.522677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.522428 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cad2c6\"" Oct 30 13:35:14.526990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.525873 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cad2c6-dockercfg-pkx89\"" Oct 30 13:35:14.531704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.531675 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk"] Oct 30 13:35:14.570943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.570907 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm"] Oct 30 13:35:14.575622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.575571 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.593836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.593808 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.593977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.593851 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqg8\" (UniqueName: \"kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8\") pod \"payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.594148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.594131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8j7\" (UniqueName: \"kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7\") pod \"payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.594280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.594259 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.594550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.594530 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm"] Oct 30 13:35:14.605686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.599966 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:14.731621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.715712 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8j7\" (UniqueName: \"kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7\") pod \"payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.731621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.726998 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.731621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.727467 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.731621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.727524 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqg8\" (UniqueName: \"kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8\") pod \"payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.731621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.727930 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.740996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.740958 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.821760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.804360 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8j7\" (UniqueName: \"kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7\") pod \"payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.846621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.837688 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:14.889617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.866840 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqg8\" (UniqueName: \"kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8\") pod \"payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:14.890470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:14.890437 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:15.207563 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:15.207530 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5156f402_98f2_4349_9956_4874b8b04ed4.slice/crio-b0f260df082ec062e0f164bb758b98e48919915c52fb109ff1b9aadffe13da84 WatchSource:0}: Error finding container b0f260df082ec062e0f164bb758b98e48919915c52fb109ff1b9aadffe13da84: Status 404 returned error can't find the container with id b0f260df082ec062e0f164bb758b98e48919915c52fb109ff1b9aadffe13da84 Oct 30 13:35:15.208137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:15.208091 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk"] Oct 30 13:35:15.244562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:15.244518 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm"] Oct 30 13:35:15.248952 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:15.248923 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29f0d1c_172c_4b0b_9de5_29bcc6a93818.slice/crio-0656b749b83f7e099d5a0dc98152572dffa5f9cd881920c249214b97ff95d468 WatchSource:0}: Error finding container 0656b749b83f7e099d5a0dc98152572dffa5f9cd881920c249214b97ff95d468: Status 404 returned error can't find the container with id 0656b749b83f7e099d5a0dc98152572dffa5f9cd881920c249214b97ff95d468 Oct 30 13:35:15.284328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:15.284236 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" event={"ID":"a29f0d1c-172c-4b0b-9de5-29bcc6a93818","Type":"ContainerStarted","Data":"0656b749b83f7e099d5a0dc98152572dffa5f9cd881920c249214b97ff95d468"} Oct 30 13:35:15.286712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:15.286683 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" event={"ID":"5156f402-98f2-4349-9956-4874b8b04ed4","Type":"ContainerStarted","Data":"b0f260df082ec062e0f164bb758b98e48919915c52fb109ff1b9aadffe13da84"} Oct 30 13:35:15.534783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:15.534684 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:16.295956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:16.295917 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" event={"ID":"a29f0d1c-172c-4b0b-9de5-29bcc6a93818","Type":"ContainerStarted","Data":"dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6"} Oct 30 13:35:16.296167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:16.295989 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:16.297687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:16.297654 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" event={"ID":"5156f402-98f2-4349-9956-4874b8b04ed4","Type":"ContainerStarted","Data":"91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0"} Oct 30 13:35:16.297842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:16.297789 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:16.335559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:16.335517 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" podStartSLOduration=2.335503985 podStartE2EDuration="2.335503985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:16.332433847 +0000 UTC m=+1892.199052020" watchObservedRunningTime="2026-10-30 13:35:16.335503985 +0000 UTC m=+1892.202122157" Oct 30 13:35:16.370037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:16.369981 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" podStartSLOduration=2.36996507 podStartE2EDuration="2.36996507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:16.368758877 +0000 UTC m=+1892.235377051" watchObservedRunningTime="2026-10-30 13:35:16.36996507 +0000 UTC m=+1892.236583245" Oct 30 13:35:17.225413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:17.225380 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:18.813288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:18.813246 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:20.529741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:20.524132 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:22.090189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:22.090155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.058133 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:23.058093 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:35:23.715515 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:23.715478 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.233588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.233511 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276"] Oct 30 13:35:25.239025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.239003 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.256631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.256591 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276"] Oct 30 13:35:25.269453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.269424 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3f24fd-serving-cert\"" Oct 30 13:35:25.372484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.372447 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.409982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.409947 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.410577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.410210 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.410577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.410315 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wv5\" (UniqueName: \"kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.410577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.410479 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.517655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.511864 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.517655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.511962 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.517655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.512182 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.517655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.512267 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.517655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.512285 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wv5\" (UniqueName: \"kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.517655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.513584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.520304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.520282 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.521233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.521209 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.532237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.532214 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wv5\" (UniqueName: \"kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5\") pod \"maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.556960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.556933 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:25.862449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:25.862418 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276"] Oct 30 13:35:25.865226 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:25.865191 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd975ade9_c5cf_4335_9d2f_3ead72ae9bb2.slice/crio-3b0d78002d90d3ae76901ee9f124d69e78682f0a33b923ed4accbd7e70fdaf79 WatchSource:0}: Error finding container 3b0d78002d90d3ae76901ee9f124d69e78682f0a33b923ed4accbd7e70fdaf79: Status 404 returned error can't find the container with id 3b0d78002d90d3ae76901ee9f124d69e78682f0a33b923ed4accbd7e70fdaf79 Oct 30 13:35:26.448550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:26.448514 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" event={"ID":"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2","Type":"ContainerStarted","Data":"3b0d78002d90d3ae76901ee9f124d69e78682f0a33b923ed4accbd7e70fdaf79"} Oct 30 13:35:27.143621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:27.141311 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:27.320213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:27.320185 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 13:35:27.320388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:27.320247 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 13:35:27.458171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:27.458078 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" event={"ID":"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2","Type":"ContainerStarted","Data":"19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275"} Oct 30 13:35:27.458692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:27.458180 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:27.498301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:27.498244 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" podStartSLOduration=2.068467036 podStartE2EDuration="2.498231332s" totalImagesPullingTime="429.764296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:27.495733778 +0000 UTC m=+1903.362351963" watchObservedRunningTime="2026-10-30 13:35:27.498231332 +0000 UTC m=+1903.364849505" Oct 30 13:35:28.786021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:28.785985 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:30.346106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:30.346060 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:31.257404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:31.257370 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:31.908151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:31.908119 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:33.470360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:33.470328 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 13:35:34.486650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:34.486590 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:34.718081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:34.718025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:35.012910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:35.012870 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx"] Oct 30 13:35:35.092944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:35.092911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:35.384928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:35.384790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:36.728354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.728317 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6"] Oct 30 13:35:36.732979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.732950 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.751529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.751499 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6"] Oct 30 13:35:36.759027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.758660 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cad2c6-serving-cert\"" Oct 30 13:35:36.834902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.834867 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.835106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.834929 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.835180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.835132 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.835263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.835244 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfks\" (UniqueName: \"kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.853849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.853821 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:36.946266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.946220 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfks\" (UniqueName: \"kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.947244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.947215 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.947490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.947472 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.947761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.947738 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.950179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.950152 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.992666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.989178 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.992666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.990817 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfks\" (UniqueName: \"kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:36.994803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:36.994766 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs\") pod \"maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:37.000384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.000335 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:37.048712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.048686 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:37.135962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.135931 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:37.315705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.308531 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:37.422968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.422103 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6"] Oct 30 13:35:37.428247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.428222 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:35:37.560683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:37.560646 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" event={"ID":"70c6f506-deae-4d9e-85c7-f4507adba4b2","Type":"ContainerStarted","Data":"441a19b833e121b1c2812cb2d4e22a06f4c78b0f791358e59f73cf12121bcee8"} Oct 30 13:35:38.570652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:38.570611 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" event={"ID":"70c6f506-deae-4d9e-85c7-f4507adba4b2","Type":"ContainerStarted","Data":"660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331"} Oct 30 13:35:38.571568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:38.571542 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:38.614555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:38.614497 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" podStartSLOduration=2.249563149 podStartE2EDuration="2.614481388s" totalImagesPullingTime="364.918239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:38.611783395 +0000 UTC m=+1914.478401570" watchObservedRunningTime="2026-10-30 13:35:38.614481388 +0000 UTC m=+1914.481099563" Oct 30 13:35:38.708380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:38.708346 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:38.848055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:38.847974 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.204234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:39.204150 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.394624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:39.388180 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:40.185626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:40.185493 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9"] Oct 30 13:35:40.346200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:40.346118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:41.043252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:41.043184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:41.800237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:41.800201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:42.192123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.192042 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp"] Oct 30 13:35:42.207647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.207588 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.216368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.216343 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp"] Oct 30 13:35:42.252226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.252191 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-310546-openshift-default-dockercfg-vt8ps\"" Oct 30 13:35:42.278654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.277066 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:42.294785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.294747 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.295086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.295061 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.295322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.295302 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.295533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.295512 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.295693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.295676 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.295919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.295895 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.296215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.296193 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.296626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.296575 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszvz\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.296799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.296780 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.399712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.399671 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.426623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.425809 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.426623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.425925 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszvz\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.426623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.425987 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.426623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.426067 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.426623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.426135 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.436068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.432222 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.442863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.442764 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.443199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.443171 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.443351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.443333 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.448674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.444524 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.468758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.451591 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.468981 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.451870 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.469077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.452514 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.469169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.457863 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.484627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.481481 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszvz\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.493720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.491162 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.502219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.502186 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token\") pod \"e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.573083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.573054 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:42.721439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.721052 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:42.991884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.991851 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp"] Oct 30 13:35:42.993308 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:35:42.993289 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f4286a_6f10_46bc_960b_ae6b3178f744.slice/crio-3290b435d57a867745021984db806beb20c3a3b10184981ec8b9395de901ccb0 WatchSource:0}: Error finding container 3290b435d57a867745021984db806beb20c3a3b10184981ec8b9395de901ccb0: Status 404 returned error can't find the container with id 3290b435d57a867745021984db806beb20c3a3b10184981ec8b9395de901ccb0 Oct 30 13:35:42.996058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.996025 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:42.996138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.996105 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:42.996174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:42.996136 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:43.169654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:43.169366 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:43.409072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:43.408991 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:43.559747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:43.553244 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:43.700983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:43.700899 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" event={"ID":"75f4286a-6f10-46bc-960b-ae6b3178f744","Type":"ContainerStarted","Data":"13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23"} Oct 30 13:35:43.700983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:43.700954 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" event={"ID":"75f4286a-6f10-46bc-960b-ae6b3178f744","Type":"ContainerStarted","Data":"3290b435d57a867745021984db806beb20c3a3b10184981ec8b9395de901ccb0"} Oct 30 13:35:43.767361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:43.766817 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podStartSLOduration=1.766801186 podStartE2EDuration="1.766801186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:43.763922715 +0000 UTC m=+1919.630540889" watchObservedRunningTime="2026-10-30 13:35:43.766801186 +0000 UTC m=+1919.633419358" Oct 30 13:35:44.574418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:44.574376 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:44.580180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:44.580149 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:44.587428 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:44.587399 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 13:35:44.715864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:44.715662 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:44.717567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:44.717127 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:35:45.155158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:45.155111 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:45.330621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:45.328534 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:46.489009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:46.488975 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:46.653044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:46.653006 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:47.288168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:47.288129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:48.136212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:48.136179 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:48.283053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:48.283023 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:49.117625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:49.107330 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:50.038114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.038064 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" podUID="7db209e0-95d6-4b63-848c-2a83c021c673" containerName="main" containerID="cri-o://67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6" gracePeriod=45 Oct 30 13:35:50.383724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.383683 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:35:50.497903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.497866 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq\" (UniqueName: \"kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq\") pod \"7db209e0-95d6-4b63-848c-2a83c021c673\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " Oct 30 13:35:50.498060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.497975 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs\" (UniqueName: \"kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs\") pod \"7db209e0-95d6-4b63-848c-2a83c021c673\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " Oct 30 13:35:50.498130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498084 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home\") pod \"7db209e0-95d6-4b63-848c-2a83c021c673\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " Oct 30 13:35:50.498130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498124 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir\") pod \"7db209e0-95d6-4b63-848c-2a83c021c673\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " Oct 30 13:35:50.498242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498182 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm\") pod \"7db209e0-95d6-4b63-848c-2a83c021c673\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " Oct 30 13:35:50.498242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498209 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache\") pod \"7db209e0-95d6-4b63-848c-2a83c021c673\" (UID: \"7db209e0-95d6-4b63-848c-2a83c021c673\") " Oct 30 13:35:50.498511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498476 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir" pod "7db209e0-95d6-4b63-848c-2a83c021c673" (UID: "7db209e0-95d6-4b63-848c-2a83c021c673"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:50.498703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498671 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache" pod "7db209e0-95d6-4b63-848c-2a83c021c673" (UID: "7db209e0-95d6-4b63-848c-2a83c021c673"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:50.498992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498968 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:50.499093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498394 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home" pod "7db209e0-95d6-4b63-848c-2a83c021c673" (UID: "7db209e0-95d6-4b63-848c-2a83c021c673"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:50.499093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.498997 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:50.500483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.500452 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs" pod "7db209e0-95d6-4b63-848c-2a83c021c673" (UID: "7db209e0-95d6-4b63-848c-2a83c021c673"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:50.500576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.500473 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm" pod "7db209e0-95d6-4b63-848c-2a83c021c673" (UID: "7db209e0-95d6-4b63-848c-2a83c021c673"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:50.500576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.500516 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq" pod "7db209e0-95d6-4b63-848c-2a83c021c673" (UID: "7db209e0-95d6-4b63-848c-2a83c021c673"). InnerVolumeSpecName "kube-api-access-ggsmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:50.600935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.600861 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggsmq\" (UniqueName: \"kubernetes.io/projected/7db209e0-95d6-4b63-848c-2a83c021c673-kube-api-access-ggsmq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:50.600935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.600893 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db209e0-95d6-4b63-848c-2a83c021c673-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:50.600935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.600905 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:50.600935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.600912 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db209e0-95d6-4b63-848c-2a83c021c673-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:50.783869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.783824 5693 generic.go:356] "Generic (PLEG): container finished" podID="7db209e0-95d6-4b63-848c-2a83c021c673" containerID="67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6" exitCode=0 Oct 30 13:35:50.784037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.783924 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" Oct 30 13:35:50.784037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.783961 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" event={"ID":"7db209e0-95d6-4b63-848c-2a83c021c673","Type":"ContainerDied","Data":"67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6"} Oct 30 13:35:50.784037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.784010 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx" event={"ID":"7db209e0-95d6-4b63-848c-2a83c021c673","Type":"ContainerDied","Data":"bab41a1f654f60daa77171fb40187e34679680bf0e1ed85961ceb280fa460026"} Oct 30 13:35:50.784037 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.784029 5693 scope.go:122] "RemoveContainer" containerID="67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6" Oct 30 13:35:50.795643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.795580 5693 scope.go:122] "RemoveContainer" containerID="67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6" Oct 30 13:35:50.795935 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:50.795905 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6\": container with ID starting with 67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6 not found: ID does not exist" containerID="67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6" Oct 30 13:35:50.796043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.795941 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6"} err="failed to get container status \"67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6\": rpc error: code = NotFound desc = could not find container \"67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6\": container with ID starting with 67a4575f930f40f68264cafee505f633a642ee5c2c636cc1df4969540085d1a6 not found: ID does not exist" Oct 30 13:35:50.862677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.861167 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx"] Oct 30 13:35:50.871648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.865946 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cab6c5/auth-test-model-cab6c5-kserve-8f8b85cbf-l9nvx"] Oct 30 13:35:50.871648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:50.866820 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:51.793351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:51.793316 5693 generic.go:356] "Generic (PLEG): container finished" podID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerID="2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244" exitCode=0 Oct 30 13:35:51.793919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:51.793391 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" event={"ID":"6770ae55-0287-4dc9-a718-01f0e2136e8c","Type":"ContainerDied","Data":"2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244"} Oct 30 13:35:51.793989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:51.793964 5693 scope.go:122] "RemoveContainer" containerID="2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244" Oct 30 13:35:51.795728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:51.795699 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:51.795868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:51.795765 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:51.795868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:51.795797 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:35:52.258521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:52.258488 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db209e0-95d6-4b63-848c-2a83c021c673" path="/var/lib/kubelet/pods/7db209e0-95d6-4b63-848c-2a83c021c673/volumes" Oct 30 13:35:52.548681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:52.547152 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:52.802375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:52.802296 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" event={"ID":"6770ae55-0287-4dc9-a718-01f0e2136e8c","Type":"ContainerStarted","Data":"c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b"} Oct 30 13:35:53.078310 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:53.078225 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:35:53.241624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:53.240963 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:35:53.241624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:53.241089 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:35:53.241624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:53.241135 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:35:54.260680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:54.258542 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:35:54.260680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:54.258667 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:35:54.361894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:54.361830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:54.499236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:54.499201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:55.052389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.052343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:55.239492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.238998 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:35:55.239492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.239076 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:35:55.253229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.253182 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" podUID="ac6dfe77-6f24-44df-9675-7ded9f7f7dca" containerName="main" containerID="cri-o://1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d" gracePeriod=45 Oct 30 13:35:55.570739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.570705 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:35:55.725815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.722557 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs\") pod \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " Oct 30 13:35:55.725815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.722632 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache\") pod \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " Oct 30 13:35:55.725815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.722773 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d\" (UniqueName: \"kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d\") pod \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " Oct 30 13:35:55.725815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.722874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm\") pod \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " Oct 30 13:35:55.725815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.722976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home\") pod \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " Oct 30 13:35:55.725815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.723028 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir\") pod \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\" (UID: \"ac6dfe77-6f24-44df-9675-7ded9f7f7dca\") " Oct 30 13:35:55.798563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.777403 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache" pod "ac6dfe77-6f24-44df-9675-7ded9f7f7dca" (UID: "ac6dfe77-6f24-44df-9675-7ded9f7f7dca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:55.802784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.802743 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home" pod "ac6dfe77-6f24-44df-9675-7ded9f7f7dca" (UID: "ac6dfe77-6f24-44df-9675-7ded9f7f7dca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:55.803055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.803026 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir" pod "ac6dfe77-6f24-44df-9675-7ded9f7f7dca" (UID: "ac6dfe77-6f24-44df-9675-7ded9f7f7dca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:55.863640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.827807 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:55.863640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.827843 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:55.863640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.827859 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:55.865892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.865792 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm" pod "ac6dfe77-6f24-44df-9675-7ded9f7f7dca" (UID: "ac6dfe77-6f24-44df-9675-7ded9f7f7dca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:55.865892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.865864 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs" pod "ac6dfe77-6f24-44df-9675-7ded9f7f7dca" (UID: "ac6dfe77-6f24-44df-9675-7ded9f7f7dca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:55.871673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.868789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d" pod "ac6dfe77-6f24-44df-9675-7ded9f7f7dca" (UID: "ac6dfe77-6f24-44df-9675-7ded9f7f7dca"). InnerVolumeSpecName "kube-api-access-8gt2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:55.873765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.873737 5693 generic.go:356] "Generic (PLEG): container finished" podID="ac6dfe77-6f24-44df-9675-7ded9f7f7dca" containerID="1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d" exitCode=0 Oct 30 13:35:55.873934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.873832 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" event={"ID":"ac6dfe77-6f24-44df-9675-7ded9f7f7dca","Type":"ContainerDied","Data":"1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d"} Oct 30 13:35:55.874029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.873960 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" event={"ID":"ac6dfe77-6f24-44df-9675-7ded9f7f7dca","Type":"ContainerDied","Data":"9634053acf3998527684d5e25a6ea22718e1a5fb4b2a66822773ff2d02eef9c4"} Oct 30 13:35:55.874029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.873987 5693 scope.go:122] "RemoveContainer" containerID="1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d" Oct 30 13:35:55.874146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.873845 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9" Oct 30 13:35:55.890220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.889827 5693 scope.go:122] "RemoveContainer" containerID="1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d" Oct 30 13:35:55.890220 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:35:55.890063 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d\": container with ID starting with 1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d not found: ID does not exist" containerID="1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d" Oct 30 13:35:55.890220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.890090 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d"} err="failed to get container status \"1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d\": rpc error: code = NotFound desc = could not find container \"1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d\": container with ID starting with 1ea94c2ebfb3e0d8bae53ff24b57ffced06784db90d03041b75b7b54e87df94d not found: ID does not exist" Oct 30 13:35:55.924936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.924897 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9"] Oct 30 13:35:55.952499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.952052 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gt2d\" (UniqueName: \"kubernetes.io/projected/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-kube-api-access-8gt2d\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:55.952499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.952094 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:55.952499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.952109 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6dfe77-6f24-44df-9675-7ded9f7f7dca-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:35:55.954130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:55.954102 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-552cc3/auth-test-model-552cc3-kserve-556df44d4f-qcxr9"] Oct 30 13:35:56.099947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:56.098018 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:56.243622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:56.238925 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:35:56.243622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:56.238998 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:35:56.262766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:56.262733 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6dfe77-6f24-44df-9675-7ded9f7f7dca" path="/var/lib/kubelet/pods/ac6dfe77-6f24-44df-9675-7ded9f7f7dca/volumes" Oct 30 13:35:56.505800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:56.505772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:57.072153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:57.072124 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:57.248682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:57.248608 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:35:57.248682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:57.248668 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:35:57.553446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:57.553398 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:57.718622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:57.718452 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:58.198648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:58.198574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:58.239167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:58.239127 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:35:58.239722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:58.239686 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:35:59.292372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:59.292338 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:35:59.293540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:59.293519 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:35:59.295931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:59.295908 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 13:35:59.906066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:35:59.906031 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:01.562248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:01.562202 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:01.739051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:01.739010 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:03.311947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:03.311904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:03.535014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:03.534983 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:03.766263 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:03.765573 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:05.335153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:05.335123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.901790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:06.901750 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:08.650450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:08.650421 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:10.456442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:10.456389 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:12.179687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:12.179654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:12.321148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:12.321113 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:13.820908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:13.820869 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:14.158304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:14.157011 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6"] Oct 30 13:36:14.158304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:14.157225 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" containerID="cri-o://89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee" gracePeriod=30 Oct 30 13:36:14.381635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:14.380416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:15.167555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.167520 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:15.563153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.563119 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg"] Oct 30 13:36:15.564555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.564521 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7db209e0-95d6-4b63-848c-2a83c021c673" containerName="main" Oct 30 13:36:15.564555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.564553 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db209e0-95d6-4b63-848c-2a83c021c673" containerName="main" Oct 30 13:36:15.564801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.564590 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ac6dfe77-6f24-44df-9675-7ded9f7f7dca" containerName="main" Oct 30 13:36:15.564801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.564613 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6dfe77-6f24-44df-9675-7ded9f7f7dca" containerName="main" Oct 30 13:36:15.564801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.564787 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6dfe77-6f24-44df-9675-7ded9f7f7dca" containerName="main" Oct 30 13:36:15.564988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.564831 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db209e0-95d6-4b63-848c-2a83c021c673" containerName="main" Oct 30 13:36:15.569782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.569758 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.585508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.585485 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-844856-openshift-default-dockercfg-wmrs2\"" Oct 30 13:36:15.589235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.589212 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg"] Oct 30 13:36:15.599669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.599635 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.599809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.599713 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.599908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.599887 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.599980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.599968 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.600143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.600121 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xcg\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.600234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.600206 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.600299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.600243 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.600366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.600301 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.600366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.600348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.657952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.657918 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:15.701200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701166 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701210 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701264 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701404 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701463 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701524 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701636 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701660 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701739 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcg\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.701954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.701756 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.702103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.702076 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.702173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.702125 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.702173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.702154 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.703932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.703905 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.704219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.704193 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.721088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.721051 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xcg\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.721343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.721320 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:15.825325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.825168 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:15.899696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:15.896999 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:16.364726 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.364686 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg"] Oct 30 13:36:16.376742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.376700 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:16.377565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.377535 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:16.377775 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.377755 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:16.464036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.464007 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:16.597019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.596983 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:16.775689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.775644 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm"] Oct 30 13:36:16.776177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.776147 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" containerID="cri-o://2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca" gracePeriod=30 Oct 30 13:36:16.810487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.804747 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:16.812788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.812760 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7683-2e37b5"] Oct 30 13:36:16.819060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.818178 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-2e37b5"] Oct 30 13:36:16.819060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.818281 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:16.851901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.851874 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4bkkd\"" Oct 30 13:36:16.986457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:16.986420 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52bp\" (UniqueName: \"kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp\") pod \"test-curl-7683-2e37b5\" (UID: \"ee4e2828-0c71-4c7b-9aed-c265693e015a\") " pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:17.088643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.088544 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52bp\" (UniqueName: \"kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp\") pod \"test-curl-7683-2e37b5\" (UID: \"ee4e2828-0c71-4c7b-9aed-c265693e015a\") " pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:17.109436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.109404 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52bp\" (UniqueName: \"kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp\") pod \"test-curl-7683-2e37b5\" (UID: \"ee4e2828-0c71-4c7b-9aed-c265693e015a\") " pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:17.195518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.192573 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:17.206891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.206857 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" event={"ID":"2a9623ae-3597-4add-97e7-a21f3b38c6b6","Type":"ContainerStarted","Data":"c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2"} Oct 30 13:36:17.207210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.207175 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" event={"ID":"2a9623ae-3597-4add-97e7-a21f3b38c6b6","Type":"ContainerStarted","Data":"50bc178d6a2519b6749589f2757944b096b3c2478c122e2f91c4f8d2b9a86f31"} Oct 30 13:36:17.559632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.558981 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podStartSLOduration=2.558956585 podStartE2EDuration="2.558956585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:17.255942965 +0000 UTC m=+1953.122561140" watchObservedRunningTime="2026-10-30 13:36:17.558956585 +0000 UTC m=+1953.425574759" Oct 30 13:36:17.562911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.562880 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-2e37b5"] Oct 30 13:36:17.897765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.897682 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:17.904074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:17.904047 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:18.216022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.215922 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-2e37b5" event={"ID":"ee4e2828-0c71-4c7b-9aed-c265693e015a","Type":"ContainerStarted","Data":"dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580"} Oct 30 13:36:18.216022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.215971 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-2e37b5" event={"ID":"ee4e2828-0c71-4c7b-9aed-c265693e015a","Type":"ContainerStarted","Data":"9caeb783da1361b524ae969061bd7e07f251cc3f06b5c04ec050303a367116d9"} Oct 30 13:36:18.216251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.216172 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:18.217109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.217083 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:36:18.265527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.261419 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:18.346621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.331295 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7683-2e37b5" podStartSLOduration=2.331273184 podStartE2EDuration="2.331273184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:18.293320768 +0000 UTC m=+1954.159938941" watchObservedRunningTime="2026-10-30 13:36:18.331273184 +0000 UTC m=+1954.197891360" Oct 30 13:36:18.346621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.341642 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k8pb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:36:18.346621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:18.341715 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:36:19.240128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.238919 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7683-2e37b5"] Oct 30 13:36:19.242364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.242337 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7683-2e37b5"] Oct 30 13:36:19.468893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.468827 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:36:19.500753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.500718 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.500944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.500831 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.500966 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501011 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:19.501091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501038 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501074 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501213 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501217 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:19.501259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501392 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501459 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs\") pod \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\" (UID: \"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06\") " Oct 30 13:36:19.501787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501755 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:19.501938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.501908 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:19.502312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.502287 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:19.502312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.502306 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.502518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.502330 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.502518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.502345 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.502518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.502358 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.504130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.503945 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:36:19.504763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.504617 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "kube-api-access-9tcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:19.506039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.506015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:19.506374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.506355 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy" pod "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" (UID: "ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:19.605347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.605308 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.605347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.605345 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.605548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.605358 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.605548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.605373 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.605548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.605386 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tcmq\" (UniqueName: \"kubernetes.io/projected/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06-kube-api-access-9tcmq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:19.726504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.726472 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:19.863014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:19.862981 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:20.176724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.167754 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:20.233786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.233752 5693 generic.go:356] "Generic (PLEG): container finished" podID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerID="89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee" exitCode=0 Oct 30 13:36:20.233975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.233802 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" Oct 30 13:36:20.233975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.233833 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" event={"ID":"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06","Type":"ContainerDied","Data":"89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee"} Oct 30 13:36:20.233975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.233890 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6" event={"ID":"ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06","Type":"ContainerDied","Data":"9a8e68f3c1942f92107e9314a01632c6399a5b82bd53aa400c65f9113fd34d2a"} Oct 30 13:36:20.233975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.233917 5693 scope.go:122] "RemoveContainer" containerID="89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee" Oct 30 13:36:20.234207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.234003 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7683-2e37b5" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" containerName="test-curl-7683-2e37b5" containerID="cri-o://dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580" gracePeriod=2 Oct 30 13:36:20.247301 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:20.247273 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7683-2e37b5\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:20.252042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.252020 5693 scope.go:122] "RemoveContainer" containerID="89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee" Oct 30 13:36:20.252416 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:20.252387 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee\": container with ID starting with 89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee not found: ID does not exist" containerID="89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee" Oct 30 13:36:20.252479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.252428 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee"} err="failed to get container status \"89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee\": rpc error: code = NotFound desc = could not find container \"89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee\": container with ID starting with 89ee1afd96d4356bc8889a66728e3c8e796b73044fdcb2a1cd599b80582018ee not found: ID does not exist" Oct 30 13:36:20.301346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.301310 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6"] Oct 30 13:36:20.312116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:20.312082 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k8pb6"] Oct 30 13:36:21.377548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:21.377498 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:21.522217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:21.522158 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:22.005586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.005165 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:36:22.044849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.044811 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.045048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.044872 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.045048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045016 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045086 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045130 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045192 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045349 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045358 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045444 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045508 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045562 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo\") pod \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\" (UID: \"188c84ed-c1ed-4846-8f5f-feb3790fb8ad\") " Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045643 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:22.046723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.045956 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:22.047127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.046322 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:22.047174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.046498 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.047174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.047158 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.047244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.047175 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.047244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.047191 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.047965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.047937 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:22.048221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.048188 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "kube-api-access-7d5tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:22.048778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.048749 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:22.048974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.048957 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo" pod "188c84ed-c1ed-4846-8f5f-feb3790fb8ad" (UID: "188c84ed-c1ed-4846-8f5f-feb3790fb8ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:36:22.102296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.102196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:22.148824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.148778 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.148824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.148816 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.148824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.148831 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5tv\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-kube-api-access-7d5tv\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.149152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.148844 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.149152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.148858 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c84ed-c1ed-4846-8f5f-feb3790fb8ad-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:22.293413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.293378 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" path="/var/lib/kubelet/pods/ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06/volumes" Oct 30 13:36:22.294103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.294077 5693 generic.go:356] "Generic (PLEG): container finished" podID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerID="2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca" exitCode=0 Oct 30 13:36:22.294103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.294090 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" event={"ID":"188c84ed-c1ed-4846-8f5f-feb3790fb8ad","Type":"ContainerDied","Data":"2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca"} Oct 30 13:36:22.294292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.294133 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" event={"ID":"188c84ed-c1ed-4846-8f5f-feb3790fb8ad","Type":"ContainerDied","Data":"faadf1b8dbe8ed727b57e0290207490b63a0920c11d2dbe5adf73368fe8bae8b"} Oct 30 13:36:22.294292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.294159 5693 scope.go:122] "RemoveContainer" containerID="2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca" Oct 30 13:36:22.294292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.294270 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm" Oct 30 13:36:22.310146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.310115 5693 scope.go:122] "RemoveContainer" containerID="2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca" Oct 30 13:36:22.310505 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:22.310481 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca\": container with ID starting with 2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca not found: ID does not exist" containerID="2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca" Oct 30 13:36:22.310654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.310512 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca"} err="failed to get container status \"2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca\": rpc error: code = NotFound desc = could not find container \"2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca\": container with ID starting with 2feaee1870768cce19b1c4cb9ef4499d35a3c27cf33795be97200d32c9b140ca not found: ID does not exist" Oct 30 13:36:22.363208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.363121 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm"] Oct 30 13:36:22.375160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.375132 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-5dmwm"] Oct 30 13:36:22.449416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.449385 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:22.475866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.475825 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp\" (UniqueName: \"kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp\") pod \"ee4e2828-0c71-4c7b-9aed-c265693e015a\" (UID: \"ee4e2828-0c71-4c7b-9aed-c265693e015a\") " Oct 30 13:36:22.477965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.477928 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp" pod "ee4e2828-0c71-4c7b-9aed-c265693e015a" (UID: "ee4e2828-0c71-4c7b-9aed-c265693e015a"). InnerVolumeSpecName "kube-api-access-h52bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:22.578304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:22.578258 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52bp\" (UniqueName: \"kubernetes.io/projected/ee4e2828-0c71-4c7b-9aed-c265693e015a-kube-api-access-h52bp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:23.057084 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:23.057042 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:36:23.308626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:23.305849 5693 generic.go:356] "Generic (PLEG): container finished" podID="ee4e2828-0c71-4c7b-9aed-c265693e015a" containerID="dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580" exitCode=137 Oct 30 13:36:23.308626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:23.305906 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:23.308626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:23.305947 5693 scope.go:122] "RemoveContainer" containerID="dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580" Oct 30 13:36:23.344019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:23.316672 5693 scope.go:122] "RemoveContainer" containerID="dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580" Oct 30 13:36:23.344019 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:23.316946 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580\": container with ID starting with dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580 not found: ID does not exist" containerID="dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580" Oct 30 13:36:23.344019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:23.316972 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580"} err="failed to get container status \"dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580\": rpc error: code = NotFound desc = could not find container \"dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580\": container with ID starting with dbc8731a9cd3138715551076de1bacbfb9db718a7f7cfb0f5766a44195402580 not found: ID does not exist" Oct 30 13:36:23.344019 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:23.328472 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7683-2e37b5\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:23.821619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:23.820908 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:24.255441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.255409 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" path="/var/lib/kubelet/pods/188c84ed-c1ed-4846-8f5f-feb3790fb8ad/volumes" Oct 30 13:36:24.255823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.255808 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" path="/var/lib/kubelet/pods/ee4e2828-0c71-4c7b-9aed-c265693e015a/volumes" Oct 30 13:36:24.263802 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:24.263776 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7683-2e37b5\" is forbidden: User \"system:node:ip-10-0-130-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-43.ec2.internal' and this object" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" pod="opendatahub/test-curl-7683-2e37b5" Oct 30 13:36:24.320559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.320517 5693 generic.go:356] "Generic (PLEG): container finished" podID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerID="3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131" exitCode=0 Oct 30 13:36:24.320782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.320694 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" event={"ID":"0512a21c-cfd6-47ec-9c02-4e2079109cc2","Type":"ContainerDied","Data":"3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131"} Oct 30 13:36:24.321237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.321190 5693 scope.go:122] "RemoveContainer" containerID="3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131" Oct 30 13:36:24.323406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.323357 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:24.323498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.323464 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:24.323549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.323515 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:24.809923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.809887 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb"] Oct 30 13:36:24.810144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:24.810103 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" containerID="cri-o://e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a" gracePeriod=30 Oct 30 13:36:25.356467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:25.356421 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" event={"ID":"0512a21c-cfd6-47ec-9c02-4e2079109cc2","Type":"ContainerStarted","Data":"7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4"} Oct 30 13:36:25.396311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:25.396276 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:26.260413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:26.260387 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:36:26.351971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:26.351935 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:36:26.353231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:26.353205 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 13:36:26.353375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:26.353261 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 13:36:27.048203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:27.048167 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:27.351897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:27.351800 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 13:36:27.351897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:27.351868 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 13:36:28.350415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:28.349933 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 13:36:28.350415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:28.350015 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 13:36:28.841190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:28.841154 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:29.352029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:29.351988 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 13:36:29.352543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:29.352058 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 13:36:30.150621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.150045 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:36:30.254348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254299 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254359 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254430 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254468 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254537 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254582 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254763 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.254896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254811 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.255069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.254900 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data\") pod \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\" (UID: \"c3cd7236-a1ac-4031-ae4c-cee28ef0d476\") " Oct 30 13:36:30.255752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.255724 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:30.255977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.255955 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:30.256340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.256318 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:30.258692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.256559 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:30.260568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.259761 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:36:30.260970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.260931 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:30.268065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.268020 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:30.269489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.269461 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:30.294213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.294176 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx" pod "c3cd7236-a1ac-4031-ae4c-cee28ef0d476" (UID: "c3cd7236-a1ac-4031-ae4c-cee28ef0d476"). InnerVolumeSpecName "kube-api-access-66qbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:30.348991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.348907 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 13:36:30.348991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.348974 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 13:36:30.357056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.356956 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.356981 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.356991 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.357000 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.357008 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.357033 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66qbx\" (UniqueName: \"kubernetes.io/projected/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-kube-api-access-66qbx\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.357069 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.357089 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.357635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.357101 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3cd7236-a1ac-4031-ae4c-cee28ef0d476-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:36:30.395035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.394997 5693 generic.go:356] "Generic (PLEG): container finished" podID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerID="e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a" exitCode=0 Oct 30 13:36:30.395210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.395054 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" Oct 30 13:36:30.395210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.395080 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" event={"ID":"c3cd7236-a1ac-4031-ae4c-cee28ef0d476","Type":"ContainerDied","Data":"e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a"} Oct 30 13:36:30.395210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.395138 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb" event={"ID":"c3cd7236-a1ac-4031-ae4c-cee28ef0d476","Type":"ContainerDied","Data":"f6ff169a63a02e2166d793d61bbc062f4491f5fd581a82ee3d9f8a78ba02e428"} Oct 30 13:36:30.395210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.395163 5693 scope.go:122] "RemoveContainer" containerID="e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a" Oct 30 13:36:30.411154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.411121 5693 scope.go:122] "RemoveContainer" containerID="e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a" Oct 30 13:36:30.414238 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:30.411570 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a\": container with ID starting with e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a not found: ID does not exist" containerID="e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a" Oct 30 13:36:30.414238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.411630 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a"} err="failed to get container status \"e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a\": rpc error: code = NotFound desc = could not find container \"e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a\": container with ID starting with e25b597537d8d4b720ea3d7f4ee2829bcae4a7f5746d68e12b126bfa3963e96a not found: ID does not exist" Oct 30 13:36:30.449352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.449312 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb"] Oct 30 13:36:30.454685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.454651 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-9wxnb"] Oct 30 13:36:30.597967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:30.597927 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:31.353819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:31.353792 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:36:31.354916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:31.354890 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 13:36:32.225940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:32.225906 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:32.298509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:32.298467 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" path="/var/lib/kubelet/pods/c3cd7236-a1ac-4031-ae4c-cee28ef0d476/volumes" Oct 30 13:36:33.654766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.654731 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7"] Oct 30 13:36:33.655564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655546 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" containerName="test-curl-7683-2e37b5" Oct 30 13:36:33.655564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655565 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" containerName="test-curl-7683-2e37b5" Oct 30 13:36:33.655731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655590 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" Oct 30 13:36:33.655731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655607 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" Oct 30 13:36:33.655731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655619 5693 cpu_manager.go:414] "containerMap: removing container" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" Oct 30 13:36:33.655731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655625 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" Oct 30 13:36:33.655731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655643 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" Oct 30 13:36:33.655731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655648 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" Oct 30 13:36:33.655919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655743 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cd7236-a1ac-4031-ae4c-cee28ef0d476" containerName="istio-proxy" Oct 30 13:36:33.655919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655756 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4e2828-0c71-4c7b-9aed-c265693e015a" containerName="test-curl-7683-2e37b5" Oct 30 13:36:33.655919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655768 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5f2cd5-06c6-4523-8cb7-3748c2fc0a06" containerName="istio-proxy" Oct 30 13:36:33.655919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.655775 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="188c84ed-c1ed-4846-8f5f-feb3790fb8ad" containerName="istio-proxy" Oct 30 13:36:33.661073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.661051 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.679643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.679614 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7"] Oct 30 13:36:33.687780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.687710 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-51dc9a-serving-cert\"" Oct 30 13:36:33.706303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.706268 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.706460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.706441 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.706530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.706511 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.706685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.706662 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vxf\" (UniqueName: \"kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.790266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.790218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:33.808628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.808582 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.808845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.808687 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.808845 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:33.808734 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-serving-cert: secret "maas-api-e2e-shared-b-w0-51dc9a-serving-cert" not found Oct 30 13:36:33.808845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.808782 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vxf\" (UniqueName: \"kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.808845 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:33.808804 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls podName:7878c52d-f84f-4d12-8bdd-937eb5a9644a nodeName:}" failed. No retries permitted until 2026-10-30 13:36:34.308782705 +0000 UTC m=+1970.175400870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls") pod "maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" (UID: "7878c52d-f84f-4d12-8bdd-937eb5a9644a") : secret "maas-api-e2e-shared-b-w0-51dc9a-serving-cert" not found Oct 30 13:36:33.809126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.808995 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.809409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.809381 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.811657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.811588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.828547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.828514 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vxf\" (UniqueName: \"kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:33.925025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:33.924888 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:34.058731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:34.058690 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:34.316766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:34.316731 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:34.319265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:34.319241 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:34.589867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:34.589778 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:34.932262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:34.932234 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7"] Oct 30 13:36:34.934016 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:36:34.933969 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7878c52d_f84f_4d12_8bdd_937eb5a9644a.slice/crio-93e439d8627ba133cac8aa949fa0b3b8fd9a191116d6b960fa29c4e8e9a6f3b5 WatchSource:0}: Error finding container 93e439d8627ba133cac8aa949fa0b3b8fd9a191116d6b960fa29c4e8e9a6f3b5: Status 404 returned error can't find the container with id 93e439d8627ba133cac8aa949fa0b3b8fd9a191116d6b960fa29c4e8e9a6f3b5 Oct 30 13:36:35.434281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:35.434244 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" event={"ID":"7878c52d-f84f-4d12-8bdd-937eb5a9644a","Type":"ContainerStarted","Data":"486d52d6f06843d4364aa99adb8611b84a30d2a85aaa8cded0767989f1c65d33"} Oct 30 13:36:35.434460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:35.434299 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" event={"ID":"7878c52d-f84f-4d12-8bdd-937eb5a9644a","Type":"ContainerStarted","Data":"93e439d8627ba133cac8aa949fa0b3b8fd9a191116d6b960fa29c4e8e9a6f3b5"} Oct 30 13:36:35.434460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:35.434322 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:35.475685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:35.475638 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" podStartSLOduration=2.14241553 podStartE2EDuration="2.475592523s" totalImagesPullingTime="333.176993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:35.474533207 +0000 UTC m=+1971.341151380" watchObservedRunningTime="2026-10-30 13:36:35.475592523 +0000 UTC m=+1971.342210695" Oct 30 13:36:35.521394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:35.521306 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:37.206695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:37.206658 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:38.873579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:38.872702 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:40.411140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:40.411102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:41.446835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:41.446804 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 13:36:42.140768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:42.140736 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:42.382704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:42.382671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:44.063060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:44.063028 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:45.848622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:45.848235 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:47.718656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:47.718620 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:48.642987 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:48.642953 5693 scope.go:122] "RemoveContainer" containerID="9bb6f86505843e978a4061970dff830546ec2d890b7ef71b4bf9e609a091c487" Oct 30 13:36:48.661391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:48.661369 5693 scope.go:122] "RemoveContainer" containerID="574e66132e5a5c77ccf81ed7e93d5dfdcdcaa1bc3cd21a13e1488c350adbb69b" Oct 30 13:36:49.369257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:49.369208 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:49.617476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:49.617402 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:50.945827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:50.945789 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:51.084954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:51.084921 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:51.422208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:51.422172 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:51.588506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:51.588463 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:52.612208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.612171 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:52.784654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.784622 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k"] Oct 30 13:36:52.790137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.790117 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.806772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.806747 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k"] Oct 30 13:36:52.812417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.812393 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-844856-serving-cert\"" Oct 30 13:36:52.814308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.814279 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtjd\" (UniqueName: \"kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.814620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.814571 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.814764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.814737 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.814901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.814881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.908158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.908129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:52.916853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.916823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.917062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.917040 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtjd\" (UniqueName: \"kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.917282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.917250 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.917380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.917370 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.917556 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:52.917534 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-serving-cert: secret "maas-api-e2e-extmod-b-844856-serving-cert" not found Oct 30 13:36:52.917671 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:52.917657 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls podName:2a1169eb-6098-414b-a78e-4b7320f35189 nodeName:}" failed. No retries permitted until 2026-10-30 13:36:53.417635452 +0000 UTC m=+1989.284253606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls") pod "maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" (UID: "2a1169eb-6098-414b-a78e-4b7320f35189") : secret "maas-api-e2e-extmod-b-844856-serving-cert" not found Oct 30 13:36:52.917757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.917741 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.919834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.919811 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:52.943010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:52.942987 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtjd\" (UniqueName: \"kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:53.057532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.057444 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:53.076785 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:36:53.076748 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:36:53.199162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.199081 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:53.431478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.431440 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:53.434094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.434061 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\") pod \"maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:53.615578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.615540 5693 generic.go:356] "Generic (PLEG): container finished" podID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerID="13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23" exitCode=0 Oct 30 13:36:53.616032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.615626 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" event={"ID":"75f4286a-6f10-46bc-960b-ae6b3178f744","Type":"ContainerDied","Data":"13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23"} Oct 30 13:36:53.616180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.616163 5693 scope.go:122] "RemoveContainer" containerID="13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23" Oct 30 13:36:53.618572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.618532 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:53.618704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.618644 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:53.618704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.618698 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:53.708930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.708904 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:53.981526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:53.980318 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:54.038972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:54.038943 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k"] Oct 30 13:36:54.042001 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:36:54.041968 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1169eb_6098_414b_a78e_4b7320f35189.slice/crio-44a8d4d2ca3a78f5e71ac4a4b07b4190c0eab17614d4bf57232a0fb61820af80 WatchSource:0}: Error finding container 44a8d4d2ca3a78f5e71ac4a4b07b4190c0eab17614d4bf57232a0fb61820af80: Status 404 returned error can't find the container with id 44a8d4d2ca3a78f5e71ac4a4b07b4190c0eab17614d4bf57232a0fb61820af80 Oct 30 13:36:54.470245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:54.467071 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:54.665072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:54.662121 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" event={"ID":"75f4286a-6f10-46bc-960b-ae6b3178f744","Type":"ContainerStarted","Data":"0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07"} Oct 30 13:36:54.694206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:54.682553 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" event={"ID":"2a1169eb-6098-414b-a78e-4b7320f35189","Type":"ContainerStarted","Data":"44a8d4d2ca3a78f5e71ac4a4b07b4190c0eab17614d4bf57232a0fb61820af80"} Oct 30 13:36:54.820635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:54.819692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:54.968136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:54.968014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:55.207665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.207631 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:55.360176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.360142 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:55.573503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.573472 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:36:55.574426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.574398 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 13:36:55.574562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.574447 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 13:36:55.692004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.691910 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" event={"ID":"2a1169eb-6098-414b-a78e-4b7320f35189","Type":"ContainerStarted","Data":"7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e"} Oct 30 13:36:55.692483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.692150 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:36:55.745944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:55.745880 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" podStartSLOduration=3.378721013 podStartE2EDuration="3.74586097s" totalImagesPullingTime="367.139957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:55.744006509 +0000 UTC m=+1991.610624682" watchObservedRunningTime="2026-10-30 13:36:55.74586097 +0000 UTC m=+1991.612479147" Oct 30 13:36:56.574044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:56.574007 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 13:36:56.574235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:56.574061 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 13:36:56.855816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:56.855625 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:57.059149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:57.059110 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:57.201127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:57.201044 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:57.573697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:57.573658 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 13:36:57.573881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:57.573713 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 13:36:58.360132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:58.360091 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:58.546623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:58.544642 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:58.573736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:58.573707 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 13:36:58.573899 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:58.573755 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 13:36:58.833034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:58.833000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:59.083738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:59.083651 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:59.574476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:59.574436 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 13:36:59.574884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:59.574507 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 13:36:59.716267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:59.716233 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:36:59.770864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:36:59.770818 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:00.158393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.158353 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z"] Oct 30 13:37:00.163435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.163405 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.196244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.193033 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z"] Oct 30 13:37:00.249179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.249150 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-671403-openshift-default-dockercfg-9rd9x\"" Oct 30 13:37:00.318401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.305772 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvvl\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.318967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.318924 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.319279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.319260 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.319479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.319461 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.319693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.319676 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.319898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.319879 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.334802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.334758 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.335088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.335069 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.335331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.335315 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.335562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.307099 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:00.436688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.436585 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvvl\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.437325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.437295 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.438259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.438240 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.438415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.438399 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.438541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.438527 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.440027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.440007 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.446205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.446180 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.446957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.446930 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.459616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.456253 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.459616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.437900 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.459616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.439588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.459616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.455983 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.459616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.446016 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.471960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.446803 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.490621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.474565 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.490894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.474656 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvvl\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.491034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.474890 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.491140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.478454 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token\") pod \"e2e-noten-671403-openshift-default-765696868c-nzr7z\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.491224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.481034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:00.578940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.578910 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:37:00.734842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.734810 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 13:37:00.789449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.789410 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:00.982576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:00.982539 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:01.125850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.125766 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:01.236653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.227217 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z"] Oct 30 13:37:01.320622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.316423 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:01.320622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.316555 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:01.320622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.316616 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:01.706502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.706418 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 13:37:01.745050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.745005 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" event={"ID":"4bc8295e-8835-48ef-af0b-a0d0b810bfad","Type":"ContainerStarted","Data":"1d982d264cdb605280dc7584f42f40fd0825cb82a79b23a874c062d053daa36f"} Oct 30 13:37:01.745269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.745066 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" event={"ID":"4bc8295e-8835-48ef-af0b-a0d0b810bfad","Type":"ContainerStarted","Data":"f75bba282843606e9fb1e27ad12310b82378908c1feeeeb9533b3c2dbf432205"} Oct 30 13:37:01.789387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.789320 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" podStartSLOduration=1.789299058 podStartE2EDuration="1.789299058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:01.78828563 +0000 UTC m=+1997.654903825" watchObservedRunningTime="2026-10-30 13:37:01.789299058 +0000 UTC m=+1997.655917233" Oct 30 13:37:01.793731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:01.789772 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:02.208477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:02.208439 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:02.354969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:02.354932 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:02.520718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:02.520683 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:02.796628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:02.796526 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:02.933632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:02.926497 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:03.759893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:03.759863 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:03.760825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:03.760807 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:04.614867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:04.614830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:06.215159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:06.215122 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:07.804887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:07.804833 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:09.427921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:09.427871 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:11.144426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:11.144388 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:13.183866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:13.183816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:13.581337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:13.581282 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:15.324735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:15.324696 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:15.546229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:15.546169 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:16.165367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:16.165332 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:16.372268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:16.372209 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:17.125228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:17.125123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:17.796647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:17.796610 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:19.041862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:19.041830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:19.479103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:19.479073 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:19.630971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:19.630939 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:20.999239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:20.999203 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:21.149115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.149084 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:21.300989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.300908 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:21.514674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.506294 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm"] Oct 30 13:37:21.551167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.551079 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm"] Oct 30 13:37:21.551670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.551635 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.569154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.569120 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-74589c\"" Oct 30 13:37:21.580312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.580278 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-74589c-dockercfg-jcvd6\"" Oct 30 13:37:21.597832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.597798 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.598128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.598103 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qn2\" (UniqueName: \"kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2\") pod \"payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.699711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.699668 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qn2\" (UniqueName: \"kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2\") pod \"payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.699909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.699752 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.700500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.700474 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.719481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.719444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qn2\" (UniqueName: \"kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2\") pod \"payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:21.744931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.744898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:21.859784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.846828 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z"] Oct 30 13:37:21.873497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.866806 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" podUID="4bc8295e-8835-48ef-af0b-a0d0b810bfad" containerName="istio-proxy" containerID="cri-o://1d982d264cdb605280dc7584f42f40fd0825cb82a79b23a874c062d053daa36f" gracePeriod=30 Oct 30 13:37:21.890815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:21.890401 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:22.075795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:22.074703 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:22.265452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:22.265374 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:22.413744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:22.413710 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:22.499945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:22.499903 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm"] Oct 30 13:37:22.504989 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:37:22.504941 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f8754c_1a5a_4ea0_86eb_ad4c31e876f5.slice/crio-c6ed2bc9f563bc3401d0b3a5c748bf6a564d58b467413babffac3596073ab82f WatchSource:0}: Error finding container c6ed2bc9f563bc3401d0b3a5c748bf6a564d58b467413babffac3596073ab82f: Status 404 returned error can't find the container with id c6ed2bc9f563bc3401d0b3a5c748bf6a564d58b467413babffac3596073ab82f Oct 30 13:37:23.016218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:23.016182 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:23.057424 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:37:23.057384 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:37:23.145443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:23.141719 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" event={"ID":"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5","Type":"ContainerStarted","Data":"1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e"} Oct 30 13:37:23.145443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:23.141774 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" event={"ID":"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5","Type":"ContainerStarted","Data":"c6ed2bc9f563bc3401d0b3a5c748bf6a564d58b467413babffac3596073ab82f"} Oct 30 13:37:23.145443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:23.141983 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:23.185358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:23.185305 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" podStartSLOduration=2.185288673 podStartE2EDuration="2.185288673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:23.181302927 +0000 UTC m=+2019.047921121" watchObservedRunningTime="2026-10-30 13:37:23.185288673 +0000 UTC m=+2019.051906874" Oct 30 13:37:24.151149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.151115 5693 generic.go:356] "Generic (PLEG): container finished" podID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerID="c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2" exitCode=0 Oct 30 13:37:24.151680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.151188 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" event={"ID":"2a9623ae-3597-4add-97e7-a21f3b38c6b6","Type":"ContainerDied","Data":"c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2"} Oct 30 13:37:24.151863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.151844 5693 scope.go:122] "RemoveContainer" containerID="c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2" Oct 30 13:37:24.153482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.153448 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:24.153625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.153533 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:24.153710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.153643 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:24.688613 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:24.688555 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:25.160567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:25.160528 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" event={"ID":"2a9623ae-3597-4add-97e7-a21f3b38c6b6","Type":"ContainerStarted","Data":"9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23"} Oct 30 13:37:25.897694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:25.897657 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:37:25.897904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:25.897881 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:37:25.897954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:25.897931 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 13:37:26.357784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:26.357746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:26.897638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:26.897575 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:37:26.897812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:26.897662 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 13:37:27.180951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.180742 5693 generic.go:356] "Generic (PLEG): container finished" podID="4bc8295e-8835-48ef-af0b-a0d0b810bfad" containerID="1d982d264cdb605280dc7584f42f40fd0825cb82a79b23a874c062d053daa36f" exitCode=0 Oct 30 13:37:27.180951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.180855 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" event={"ID":"4bc8295e-8835-48ef-af0b-a0d0b810bfad","Type":"ContainerDied","Data":"1d982d264cdb605280dc7584f42f40fd0825cb82a79b23a874c062d053daa36f"} Oct 30 13:37:27.274443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.274408 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:27.386046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386008 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386066 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386108 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386147 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386258 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386340 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386374 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386520 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386569 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl\") pod \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\" (UID: \"4bc8295e-8835-48ef-af0b-a0d0b810bfad\") " Oct 30 13:37:27.386953 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.386925 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:27.387651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.387301 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.387651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.387505 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.387651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.387546 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.387651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.387570 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.387933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.387838 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.388373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.388350 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.389965 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.389921 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:27.390933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.390906 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "kube-api-access-kjvvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:27.391410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.391387 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:27.391674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.391646 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo" pod "4bc8295e-8835-48ef-af0b-a0d0b810bfad" (UID: "4bc8295e-8835-48ef-af0b-a0d0b810bfad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:27.458693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.450062 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:27.488800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488764 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.488800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488795 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.488800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488804 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.489044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488812 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.489044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488823 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.489044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488831 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc8295e-8835-48ef-af0b-a0d0b810bfad-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.489044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.488840 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjvvl\" (UniqueName: \"kubernetes.io/projected/4bc8295e-8835-48ef-af0b-a0d0b810bfad-kube-api-access-kjvvl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:37:27.897593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.897555 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:37:27.897834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:27.897637 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 13:37:28.057211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.057178 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:28.192281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.191226 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" event={"ID":"4bc8295e-8835-48ef-af0b-a0d0b810bfad","Type":"ContainerDied","Data":"f75bba282843606e9fb1e27ad12310b82378908c1feeeeb9533b3c2dbf432205"} Oct 30 13:37:28.192281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.191253 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z" Oct 30 13:37:28.192281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.191291 5693 scope.go:122] "RemoveContainer" containerID="1d982d264cdb605280dc7584f42f40fd0825cb82a79b23a874c062d053daa36f" Oct 30 13:37:28.269886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.269858 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z"] Oct 30 13:37:28.269886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.269895 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-671403-openshift-default-765696868c-nzr7z"] Oct 30 13:37:28.897751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.897719 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:37:28.898272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:28.897779 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 13:37:29.740090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:29.740057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:29.898128 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:29.898096 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 13:37:29.898545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:29.898148 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 13:37:30.255861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:30.255823 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc8295e-8835-48ef-af0b-a0d0b810bfad" path="/var/lib/kubelet/pods/4bc8295e-8835-48ef-af0b-a0d0b810bfad/volumes" Oct 30 13:37:30.902478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:30.902448 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:37:30.902978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:30.902831 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:37:30.903483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:30.903453 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 13:37:31.384063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.384027 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5"] Oct 30 13:37:31.384885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.384857 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4bc8295e-8835-48ef-af0b-a0d0b810bfad" containerName="istio-proxy" Oct 30 13:37:31.384885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.384880 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc8295e-8835-48ef-af0b-a0d0b810bfad" containerName="istio-proxy" Oct 30 13:37:31.385110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.385088 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc8295e-8835-48ef-af0b-a0d0b810bfad" containerName="istio-proxy" Oct 30 13:37:31.390452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.390430 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.405058 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.405030 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d2cecb-serving-cert\"" Oct 30 13:37:31.408874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.408392 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5"] Oct 30 13:37:31.486683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.486649 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.487070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.487037 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thvv\" (UniqueName: \"kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.487201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.487090 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.487201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.487171 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.506837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.506804 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:31.588511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.588474 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.588721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.588520 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.588721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.588653 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.588878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.588861 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thvv\" (UniqueName: \"kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.589419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.589391 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.591129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.591104 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.591319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.591298 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.608886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.608858 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thvv\" (UniqueName: \"kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv\") pod \"maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.663691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.661112 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:31.713067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.713020 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:31.798027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:31.797949 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:32.024152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:32.024122 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5"] Oct 30 13:37:32.026028 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:37:32.025991 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fca334_9e80_4517_a6be_dfe5c436d50b.slice/crio-7a793fb921b9b8e14b70db2428c1d536c08a91b80de35debc7c57ba4b695578e WatchSource:0}: Error finding container 7a793fb921b9b8e14b70db2428c1d536c08a91b80de35debc7c57ba4b695578e: Status 404 returned error can't find the container with id 7a793fb921b9b8e14b70db2428c1d536c08a91b80de35debc7c57ba4b695578e Oct 30 13:37:32.255303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:32.255269 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" event={"ID":"73fca334-9e80-4517-a6be-dfe5c436d50b","Type":"ContainerStarted","Data":"7a793fb921b9b8e14b70db2428c1d536c08a91b80de35debc7c57ba4b695578e"} Oct 30 13:37:33.257126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:33.257092 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" event={"ID":"73fca334-9e80-4517-a6be-dfe5c436d50b","Type":"ContainerStarted","Data":"1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634"} Oct 30 13:37:33.257563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:33.257228 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:33.298609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:33.298541 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" podStartSLOduration=1.979360685 podStartE2EDuration="2.298524613s" totalImagesPullingTime="319.163928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:33.295357325 +0000 UTC m=+2029.161975501" watchObservedRunningTime="2026-10-30 13:37:33.298524613 +0000 UTC m=+2029.165142787" Oct 30 13:37:33.389288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:33.389248 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:34.154425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:34.154392 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 13:37:35.059029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:35.058974 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.190966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:36.190916 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.631611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:36.631566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.798715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:36.798678 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:37.318311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:37.318278 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:37.469655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:37.469575 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:37.613576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:37.613466 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.270614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:39.270577 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 13:37:39.337928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:39.337886 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.669552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:39.669460 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:40.609056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:40.609018 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:42.299935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:42.299892 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:43.874643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:43.874557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:45.527889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:45.527845 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:47.161110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:47.160526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:48.767330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:48.767296 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:50.418132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:50.418095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:51.997889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:51.997836 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:53.056545 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:37:53.056507 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:37:53.554980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:53.554924 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:55.191176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:55.191138 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:55.332837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:55.332791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:56.744569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.744532 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg"] Oct 30 13:37:56.749707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.749685 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.768074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.768047 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg"] Oct 30 13:37:56.774828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.774797 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-74589c-serving-cert\"" Oct 30 13:37:56.777318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.777284 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.777460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.777330 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.777530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.777466 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.778015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.777554 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9zj\" (UniqueName: \"kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.871295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.871262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:56.878584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.878558 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.878768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.878588 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.878768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.878648 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.878768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.878695 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9zj\" (UniqueName: \"kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.879355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.879331 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.881340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.881317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.881534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.881515 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:56.900207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:56.900182 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9zj\" (UniqueName: \"kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj\") pod \"maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:57.010136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:57.010057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:57.064494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:57.064460 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:57.577201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:57.577171 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg"] Oct 30 13:37:57.579458 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:37:57.579431 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d025ec_a9d7_4b3b_884d_80e52a801a92.slice/crio-6c857c9fb42d17f89a0667a5ed7e0b2cc073d3ed7ab13c100bb12c819057cdba WatchSource:0}: Error finding container 6c857c9fb42d17f89a0667a5ed7e0b2cc073d3ed7ab13c100bb12c819057cdba: Status 404 returned error can't find the container with id 6c857c9fb42d17f89a0667a5ed7e0b2cc073d3ed7ab13c100bb12c819057cdba Oct 30 13:37:58.525931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:58.525897 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" event={"ID":"98d025ec-a9d7-4b3b-884d-80e52a801a92","Type":"ContainerStarted","Data":"66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395"} Oct 30 13:37:58.526321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:58.525972 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" event={"ID":"98d025ec-a9d7-4b3b-884d-80e52a801a92","Type":"ContainerStarted","Data":"6c857c9fb42d17f89a0667a5ed7e0b2cc073d3ed7ab13c100bb12c819057cdba"} Oct 30 13:37:58.526773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:58.526742 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:37:58.566747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:58.566698 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" podStartSLOduration=2.120628764 podStartE2EDuration="2.566683692s" totalImagesPullingTime="446.054928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:58.564688459 +0000 UTC m=+2054.431306631" watchObservedRunningTime="2026-10-30 13:37:58.566683692 +0000 UTC m=+2054.433301892" Oct 30 13:37:58.721947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:37:58.721910 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:00.349547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:00.349416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:01.906042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:01.906005 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:03.172271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:03.172226 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:03.312525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:03.312494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:03.587905 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:03.587865 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:05.105421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:05.105384 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:05.264088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:05.262022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:05.546975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:05.546947 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 13:38:06.709330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:06.709290 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:06.856886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:06.856851 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:07.186893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:07.186859 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:07.520799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:07.520763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:08.490616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:08.490546 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:10.436655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:10.436568 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:11.090646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:11.090581 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:11.288280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:11.288252 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:12.354620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:12.354166 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:13.691483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:13.691442 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:13.838623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:13.838574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:14.297745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:14.297709 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:15.932733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:15.932693 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:16.958620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:16.958063 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:17.709063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:17.709030 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.090879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:18.090844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.238129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:18.238094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.386666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:18.386512 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.674698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:18.674590 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.882672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:18.882625 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:19.065146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:19.065113 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:19.452757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:19.452667 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:19.612861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:19.612831 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:19.887297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:19.887261 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:20.039415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:20.039285 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:20.793243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:20.793207 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:21.260790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:21.259297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:21.414531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:21.414495 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:21.567040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:21.566963 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:21.723309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:21.723280 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:21.871543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:21.871448 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.042225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:22.042193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.197556 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:22.197474 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.644192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:22.644157 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.794431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:22.794402 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.082943 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:38:23.082909 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:38:23.680448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:23.680411 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.934946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:23.934835 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:25.432211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:25.421294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:27.073628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:27.069037 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:28.708291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:28.708253 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:30.310869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:30.310831 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:31.914628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:31.914558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:32.542240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:32.542197 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:32.679370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:32.679337 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:33.538622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:33.538150 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:35.153139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:35.153099 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:36.756614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:36.756559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:38.409243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:38.409210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:40.208438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:40.208400 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:42.333022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:42.332945 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:42.472724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:42.472685 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:42.708276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:42.708188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:43.284038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:43.283997 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:43.428056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:43.427981 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:43.571006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:43.570920 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:45.217777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:45.217744 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:46.941227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:46.941188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:47.353608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:47.353561 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:47.502307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:47.502266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:48.806043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:48.806005 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:48.985831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:48.985790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:49.135651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:49.135551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:50.556259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:50.556224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:50.728947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:50.728911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:50.897426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:50.897225 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:51.039552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:51.039514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:51.621515 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:51.621484 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:51.768254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:51.768220 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:52.439176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:52.439146 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:52.584229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:52.584197 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:52.735618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:52.730801 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:53.057771 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:38:53.057695 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:38:53.485572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.485542 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz"] Oct 30 13:38:53.490504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.490481 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.505452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.505422 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-62b7f7e20837\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.517528 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"routing-overlay\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.517536 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-e2e--cbdef895-dockercfg-rrsgw\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.517577 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"openshift-service-ca.crt\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.517659 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-plugins-e2e-extmod-b-844856\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.517691 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"kube-root-ca.crt\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.517714 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-006c41e31efa\"" Oct 30 13:38:53.526638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.522120 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz"] Oct 30 13:38:53.650450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.650415 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.650688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.650587 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.650688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.650659 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drmqb\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.650798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.650705 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.752068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.751978 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.752529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.752123 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.752529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.752159 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmqb\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.752529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.752296 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.752932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.752906 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.753017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.753001 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.754709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.754686 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.771522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.771496 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmqb\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:53.804817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:53.804793 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:54.108721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:54.108695 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz"] Oct 30 13:38:54.111739 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:38:54.111713 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a59ef4d_ab93_4411_a5c1_dd8b6dba4b97.slice/crio-e0d930ff578725f6b15af89670c1d9a052ad0c9dc61ca95b8bb42ce97f1827d9 WatchSource:0}: Error finding container e0d930ff578725f6b15af89670c1d9a052ad0c9dc61ca95b8bb42ce97f1827d9: Status 404 returned error can't find the container with id e0d930ff578725f6b15af89670c1d9a052ad0c9dc61ca95b8bb42ce97f1827d9 Oct 30 13:38:54.526395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:54.525202 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:55.126179 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:55.126147 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" event={"ID":"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97","Type":"ContainerStarted","Data":"c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647"} Oct 30 13:38:55.126587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:55.126205 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" event={"ID":"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97","Type":"ContainerStarted","Data":"e0d930ff578725f6b15af89670c1d9a052ad0c9dc61ca95b8bb42ce97f1827d9"} Oct 30 13:38:55.126587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:55.126235 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:38:55.167548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:55.167489 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" podStartSLOduration=2.167472934 podStartE2EDuration="2.167472934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:55.164554792 +0000 UTC m=+2111.031172967" watchObservedRunningTime="2026-10-30 13:38:55.167472934 +0000 UTC m=+2111.034091106" Oct 30 13:38:55.447796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:55.447707 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:56.211733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:56.211701 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:57.750572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:57.750518 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:59.382742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:38:59.382710 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:00.943240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:00.943192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:01.653946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:01.653912 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:02.628769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:02.628718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:03.512804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:03.512767 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:04.216193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:04.216161 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:05.965027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:05.964991 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:06.138184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:06.138157 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:39:07.669106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:07.669055 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:09.416848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:09.416759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:11.100381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:11.100303 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:12.699463 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:12.699383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:14.306755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:14.306718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:15.971997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:15.971944 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:17.707857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:17.707820 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:19.349776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:19.349737 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:20.941786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:20.941744 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:22.562620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:22.560473 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:22.858176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:22.858091 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:22.993588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:22.993560 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:23.057673 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:39:23.057639 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:39:24.095395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:24.095350 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:25.672801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:25.672765 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.255368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.255321 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c"] Oct 30 13:39:27.261925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.261893 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.285005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.284973 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c"] Oct 30 13:39:27.288319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.288290 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-98f1cc-serving-cert\"" Oct 30 13:39:27.384906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.384871 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.385112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.385088 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.385271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.385247 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pgz\" (UniqueName: \"kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.385366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.385349 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.387056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.387034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.490621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.488753 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pgz\" (UniqueName: \"kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.490621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.488857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.490621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.488996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.490621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.489176 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.490621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.490561 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.502433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.502402 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.533782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.519672 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.533782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.532920 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pgz\" (UniqueName: \"kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz\") pod \"maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.548754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.548654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.587150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.587112 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:27.903560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:27.903529 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c"] Oct 30 13:39:27.905854 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:39:27.905821 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b98b358_3b68_43b6_93db_d585fb5a2d82.slice/crio-dbb217b30409e5aafecd3c36bc6d270f0714a7563392e7e4e9d56d4ca25315a5 WatchSource:0}: Error finding container dbb217b30409e5aafecd3c36bc6d270f0714a7563392e7e4e9d56d4ca25315a5: Status 404 returned error can't find the container with id dbb217b30409e5aafecd3c36bc6d270f0714a7563392e7e4e9d56d4ca25315a5 Oct 30 13:39:28.522054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:28.522007 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" event={"ID":"7b98b358-3b68-43b6-93db-d585fb5a2d82","Type":"ContainerStarted","Data":"1ab136504c7126b7c0c04e446f1cd1fb61fca0d53427e7dd59bef739a85352d4"} Oct 30 13:39:28.522545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:28.522076 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" event={"ID":"7b98b358-3b68-43b6-93db-d585fb5a2d82","Type":"ContainerStarted","Data":"dbb217b30409e5aafecd3c36bc6d270f0714a7563392e7e4e9d56d4ca25315a5"} Oct 30 13:39:28.523034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:28.523010 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:28.564473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:28.564421 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" podStartSLOduration=1.154977538 podStartE2EDuration="1.564405933s" totalImagesPullingTime="409.428395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:28.561117773 +0000 UTC m=+2144.427735960" watchObservedRunningTime="2026-10-30 13:39:28.564405933 +0000 UTC m=+2144.431024105" Oct 30 13:39:29.156760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:29.156723 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:30.786129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:30.786096 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:32.391006 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:32.390968 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:33.965360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:33.965331 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:35.543123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:35.543088 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 13:39:35.696504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:35.696475 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:37.308939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:37.308886 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:38.027827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:38.027791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:39.025320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:39.025285 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:39.406154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:39.406071 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:39.870849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:39.870790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.087352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:40.087305 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.586804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:40.586751 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.745255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:40.745210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.913153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:40.913057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:41.157351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:41.157319 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:41.313790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:41.313758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.007785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:43.007753 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.247389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:43.247354 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.398521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:43.398436 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.593959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:43.593924 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.770631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:43.753876 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:44.256809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:44.256721 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:44.415375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:44.415340 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:45.368636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:45.368592 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:46.534061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.534029 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:46.636302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.636265 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5"] Oct 30 13:39:46.641115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.641094 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.660827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.660795 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5"] Oct 30 13:39:46.686183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.686141 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:46.752010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.751974 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a75d02-dockercfg-svs5v\"" Oct 30 13:39:46.752354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.752330 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a75d02\"" Oct 30 13:39:46.774741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.769823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.774741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.770037 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2blf\" (UniqueName: \"kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf\") pod \"payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.889725 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.880513 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2blf\" (UniqueName: \"kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf\") pod \"payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.890979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.890946 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.892304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.892099 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.948911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.948877 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2blf\" (UniqueName: \"kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf\") pod \"payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:46.962395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:46.962365 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:47.256418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:47.256374 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:47.319715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:47.319683 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5"] Oct 30 13:39:47.321041 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:39:47.321015 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0fbda9_c630_43fa_ab55_6d6275a24187.slice/crio-98194aee8678f9870f41d9e85cc70c7135f3d927e0bd24095b521114ff0a2b35 WatchSource:0}: Error finding container 98194aee8678f9870f41d9e85cc70c7135f3d927e0bd24095b521114ff0a2b35: Status 404 returned error can't find the container with id 98194aee8678f9870f41d9e85cc70c7135f3d927e0bd24095b521114ff0a2b35 Oct 30 13:39:47.989344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:47.989307 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" event={"ID":"1d0fbda9-c630-43fa-ab55-6d6275a24187","Type":"ContainerStarted","Data":"b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971"} Oct 30 13:39:47.989808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:47.989362 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" event={"ID":"1d0fbda9-c630-43fa-ab55-6d6275a24187","Type":"ContainerStarted","Data":"98194aee8678f9870f41d9e85cc70c7135f3d927e0bd24095b521114ff0a2b35"} Oct 30 13:39:47.989808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:47.989407 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:39:48.030577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:48.030526 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" podStartSLOduration=2.030511723 podStartE2EDuration="2.030511723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:48.026850985 +0000 UTC m=+2163.893469158" watchObservedRunningTime="2026-10-30 13:39:48.030511723 +0000 UTC m=+2163.897129914" Oct 30 13:39:48.949636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:48.944045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:50.512983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:50.512950 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:52.077505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:52.077464 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:53.056892 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:39:53.056854 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:39:53.638452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:53.638416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:55.240435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:55.240242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:55.559364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:55.559258 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:55.583327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:55.583291 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz"] Oct 30 13:39:56.850495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:56.850422 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:58.529207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:58.529172 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:59.004522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:39:59.004489 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 13:40:00.215026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.214998 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:00.584422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.584386 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" podUID="9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" containerName="payload-processing" containerID="cri-o://c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647" gracePeriod=25 Oct 30 13:40:00.881654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.881627 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:40:00.958273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.958242 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials\") pod \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " Oct 30 13:40:00.958461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.958304 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb\") pod \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " Oct 30 13:40:00.958565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.958553 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume\") pod \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " Oct 30 13:40:00.958648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.958587 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume\") pod \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\" (UID: \"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97\") " Oct 30 13:40:00.959152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.959105 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume" pod "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" (UID: "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:00.959278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.959150 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume" pod "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" (UID: "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:00.959356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.959309 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:40:00.959356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.959334 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:40:00.961017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.960985 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb" pod "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" (UID: "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97"). InnerVolumeSpecName "kube-api-access-drmqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:00.961141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:00.961116 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials" pod "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" (UID: "9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:01.060668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.060591 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:40:01.060668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.060659 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drmqb\" (UniqueName: \"kubernetes.io/projected/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97-kube-api-access-drmqb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:40:01.095911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.095874 5693 generic.go:356] "Generic (PLEG): container finished" podID="9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" containerID="c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647" exitCode=0 Oct 30 13:40:01.096158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.095921 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" Oct 30 13:40:01.096158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.095925 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" event={"ID":"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97","Type":"ContainerDied","Data":"c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647"} Oct 30 13:40:01.096158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.096058 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz" event={"ID":"9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97","Type":"ContainerDied","Data":"e0d930ff578725f6b15af89670c1d9a052ad0c9dc61ca95b8bb42ce97f1827d9"} Oct 30 13:40:01.096158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.096098 5693 scope.go:122] "RemoveContainer" containerID="c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647" Oct 30 13:40:01.113854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.113832 5693 scope.go:122] "RemoveContainer" containerID="c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647" Oct 30 13:40:01.114193 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:40:01.114169 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647\": container with ID starting with c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647 not found: ID does not exist" containerID="c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647" Oct 30 13:40:01.114291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.114204 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647"} err="failed to get container status \"c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647\": rpc error: code = NotFound desc = could not find container \"c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647\": container with ID starting with c1c7e9cbb31aa555b9664575ac2a5095c97ec1376e2a0abb64bcbad6502b2647 not found: ID does not exist" Oct 30 13:40:01.148052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.148015 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz"] Oct 30 13:40:01.151454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.151424 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cgg7kz"] Oct 30 13:40:01.849424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:01.849392 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:02.255529 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:02.255496 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" path="/var/lib/kubelet/pods/9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97/volumes" Oct 30 13:40:03.638538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:03.635117 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:05.238057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:05.238019 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:06.756203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:06.756170 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:08.354285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:08.354196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:10.006958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:10.006911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:11.607837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:11.607801 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:13.373363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:13.373322 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:14.389820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:14.389777 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:15.192642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:15.188204 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:15.665479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:15.665450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:15.864254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:15.864212 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:16.064013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:16.063963 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:16.250978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:16.250724 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:16.426969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:16.426888 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:17.061823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:17.061765 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:17.330673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:17.329930 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:17.505052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:17.505016 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:19.133887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:19.133835 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:20.713532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:20.713496 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:22.317938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:22.317901 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:23.057213 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:40:23.057167 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:40:23.883064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:23.883034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:25.483334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:25.483297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:25.622278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:25.622241 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:27.241042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:27.240997 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:28.916243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:28.916200 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:29.331733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:29.331691 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:29.464112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:29.464077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:29.935288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:29.935258 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:30.687238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:30.687198 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:32.534420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:32.534384 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:33.031930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:33.031890 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:33.570408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:33.570378 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:34.494476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:34.494444 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:34.674533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:34.674494 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:34.824881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:34.824794 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:35.629485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:35.629450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:36.332124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.332093 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf"] Oct 30 13:40:36.332942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.332923 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" containerName="payload-processing" Oct 30 13:40:36.332942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.332942 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" containerName="payload-processing" Oct 30 13:40:36.333096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.333086 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a59ef4d-ab93-4411-a5c1-dd8b6dba4b97" containerName="payload-processing" Oct 30 13:40:36.338149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.338131 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.354559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.354535 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3eb98\"" Oct 30 13:40:36.355376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.355350 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf"] Oct 30 13:40:36.363053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.363026 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"kube-root-ca.crt\"" Oct 30 13:40:36.363202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.363118 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"openshift-service-ca.crt\"" Oct 30 13:40:36.363202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.363161 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-94a5564a5433\"" Oct 30 13:40:36.363202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.363164 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"routing-overlay\"" Oct 30 13:40:36.363202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.363185 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-98ec1e033fce\"" Oct 30 13:40:36.363374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.363205 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-6acca70650d1\"" Oct 30 13:40:36.366699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.366666 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-e2e--2547b850-dockercfg-rj6kb\"" Oct 30 13:40:36.400097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.400064 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.400323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.400202 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.400323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.400245 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq22\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.400540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.400519 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.502033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.502001 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.502220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.502042 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq22\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.502266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.502252 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.502317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.502303 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.503018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.502991 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.503153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.503066 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.505185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.505154 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.523510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.523472 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq22\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.595240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.595159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:36.657923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.657880 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:36.746133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.743834 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:36.876560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:36.876489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:37.044184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:37.042543 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf"] Oct 30 13:40:37.416465 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:37.416420 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" event={"ID":"207fcd02-550e-4270-ad9b-654511d09553","Type":"ContainerStarted","Data":"a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4"} Oct 30 13:40:37.416927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:37.416495 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" event={"ID":"207fcd02-550e-4270-ad9b-654511d09553","Type":"ContainerStarted","Data":"3ccfab12f524ad660ae09ee718031469b84e04c7047b6b2fb6f90b1be9e4b08b"} Oct 30 13:40:37.416927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:37.416531 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:37.457115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:37.457063 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" podStartSLOduration=1.4570493039999999 podStartE2EDuration="1.457049304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:37.454665821 +0000 UTC m=+2213.321283995" watchObservedRunningTime="2026-10-30 13:40:37.457049304 +0000 UTC m=+2213.323667477" Oct 30 13:40:38.310331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:38.310298 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:39.960335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:39.960295 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:40.310409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:40.310368 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:40.474156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:40.474122 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:40.855521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:40.855493 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:41.796450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:41.796413 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:42.629749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:42.629714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:43.524749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:43.524715 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:43.690184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:43.690149 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:43.844803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:43.844713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:44.068660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:44.068626 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:44.351413 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:44.351364 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:44.503408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:44.503368 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:45.073231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:45.073192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:45.223697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:45.223662 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:45.814989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:45.814955 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:46.301373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:46.301342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:47.433242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:47.417177 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:47.574964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:47.574929 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:47.745558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:47.736376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:47.955636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:47.948816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:48.230373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:48.229464 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:48.429763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:48.429733 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:40:49.505912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:49.505878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:51.049523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:51.049488 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:52.634331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:52.627806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:53.056715 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:40:53.056673 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:40:53.243621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:53.238956 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:53.382490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:53.382410 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:54.526459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:54.526412 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:55.038448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:55.038414 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:55.668910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:55.668877 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:56.109752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:56.109718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:57.739867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:57.739822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:59.414148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:40:59.414102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:01.066340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:01.066302 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:01.209430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:01.209391 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:01.937480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:01.937447 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv"] Oct 30 13:41:01.937715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:01.937695 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" containerID="cri-o://3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e" gracePeriod=30 Oct 30 13:41:02.746838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:02.746806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:04.405164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:04.405130 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:06.145628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:06.136442 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:06.417245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:06.417144 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:41:06.417245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:06.417224 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:41:07.216801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.216738 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:41:07.284852 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.284812 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.284878 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.284937 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285088 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285122 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285138 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:07.285185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285178 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285298 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:07.285409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285337 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285434 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285538 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285466 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy\") pod \"96219367-d332-4190-b714-4d319878b0cb\" (UID: \"96219367-d332-4190-b714-4d319878b0cb\") " Oct 30 13:41:07.285715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.285556 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:07.286176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.286129 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:07.286176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.286165 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.286391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.286187 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.286391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.286203 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.286652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.286616 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:07.289017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.288989 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "kube-api-access-tcpzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:07.289547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.289519 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:07.289696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.289574 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:07.291164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.291135 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token" pod "96219367-d332-4190-b714-4d319878b0cb" (UID: "96219367-d332-4190-b714-4d319878b0cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:07.387997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.387965 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96219367-d332-4190-b714-4d319878b0cb-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.387997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.387995 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.388252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.388031 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.388252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.388042 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcpzc\" (UniqueName: \"kubernetes.io/projected/96219367-d332-4190-b714-4d319878b0cb-kube-api-access-tcpzc\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.388252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.388051 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96219367-d332-4190-b714-4d319878b0cb-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.388252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.388059 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96219367-d332-4190-b714-4d319878b0cb-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.770620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.762263 5693 generic.go:356] "Generic (PLEG): container finished" podID="96219367-d332-4190-b714-4d319878b0cb" containerID="3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e" exitCode=0 Oct 30 13:41:07.770620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.762469 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" event={"ID":"96219367-d332-4190-b714-4d319878b0cb","Type":"ContainerDied","Data":"3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e"} Oct 30 13:41:07.770620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.762521 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" event={"ID":"96219367-d332-4190-b714-4d319878b0cb","Type":"ContainerDied","Data":"d4eb61259562af64aa93f3aa3fb4a94dda67d5decd1e1cff0236368593de7efc"} Oct 30 13:41:07.770620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.762550 5693 scope.go:122] "RemoveContainer" containerID="3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e" Oct 30 13:41:07.770620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.762761 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv" Oct 30 13:41:07.849182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.849155 5693 scope.go:122] "RemoveContainer" containerID="3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e" Oct 30 13:41:07.854406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.854381 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv"] Oct 30 13:41:07.855118 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:41:07.855098 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e\": container with ID starting with 3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e not found: ID does not exist" containerID="3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e" Oct 30 13:41:07.855253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.855235 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e"} err="failed to get container status \"3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e\": rpc error: code = NotFound desc = could not find container \"3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e\": container with ID starting with 3a5d4947c4ce428f3ac33e135cbed419026b955a278ed9a07da87407a1e2dc1e not found: ID does not exist" Oct 30 13:41:07.859663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:07.859642 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gq6lv"] Oct 30 13:41:08.101817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:08.101713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:08.289631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:08.288224 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96219367-d332-4190-b714-4d319878b0cb" path="/var/lib/kubelet/pods/96219367-d332-4190-b714-4d319878b0cb/volumes" Oct 30 13:41:09.733621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:09.727856 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:11.479622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:11.456508 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:11.861418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:11.861319 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:12.509041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.509006 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq"] Oct 30 13:41:12.509840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.509824 5693 cpu_manager.go:414] "containerMap: removing container" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" Oct 30 13:41:12.509919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.509842 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" Oct 30 13:41:12.509970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.509961 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="96219367-d332-4190-b714-4d319878b0cb" containerName="istio-proxy" Oct 30 13:41:12.513995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.513969 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.532668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.532633 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.532833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.532739 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68m9\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.532833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.532821 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.532959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.532904 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.533248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.533225 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq"] Oct 30 13:41:12.537705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.537676 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"openshift-service-ca.crt\"" Oct 30 13:41:12.537705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.537705 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"kube-root-ca.crt\"" Oct 30 13:41:12.537903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.537725 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"routing-overlay\"" Oct 30 13:41:12.537903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.537743 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2cecb\"" Oct 30 13:41:12.577841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.577810 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-48f199422eae\"" Oct 30 13:41:12.578041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.578019 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-e2e--cb40fe81-dockercfg-5rb7k\"" Oct 30 13:41:12.578041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.578024 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-0f92d2a185e9\"" Oct 30 13:41:12.634163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.634123 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.634368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.634217 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68m9\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.634368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.634351 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.634531 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.634402 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.635055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.635030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.635589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.635568 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.637063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.637037 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.652695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.652671 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68m9\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.831944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.831871 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:12.927941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.927906 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4"] Oct 30 13:41:12.928178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:12.928152 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" containerID="cri-o://cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a" gracePeriod=30 Oct 30 13:41:13.138675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:13.138641 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq"] Oct 30 13:41:13.141264 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:41:13.141212 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f8404f_93e6_4565_91dd_32ef601f33f6.slice/crio-76e46f818bb056bfc03384f6f9bddb8617790a3431e796dc8f02f055f1f136d8 WatchSource:0}: Error finding container 76e46f818bb056bfc03384f6f9bddb8617790a3431e796dc8f02f055f1f136d8: Status 404 returned error can't find the container with id 76e46f818bb056bfc03384f6f9bddb8617790a3431e796dc8f02f055f1f136d8 Oct 30 13:41:13.554647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:13.544051 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:13.809376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:13.809336 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" event={"ID":"06f8404f-93e6-4565-91dd-32ef601f33f6","Type":"ContainerStarted","Data":"c94c944f1bb27de178d3841b6bddb234261468300944a2ab2bdeb339b946eee3"} Oct 30 13:41:13.809539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:13.809441 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:13.809539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:13.809465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" event={"ID":"06f8404f-93e6-4565-91dd-32ef601f33f6","Type":"ContainerStarted","Data":"76e46f818bb056bfc03384f6f9bddb8617790a3431e796dc8f02f055f1f136d8"} Oct 30 13:41:13.848384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:13.848329 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" podStartSLOduration=1.848317136 podStartE2EDuration="1.848317136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:13.847870874 +0000 UTC m=+2249.714489047" watchObservedRunningTime="2026-10-30 13:41:13.848317136 +0000 UTC m=+2249.714935309" Oct 30 13:41:15.302380 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:15.297329 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:16.082190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:16.082153 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:16.829635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:16.824886 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.285918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:17.285885 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.437106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:17.437074 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.686504 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:17.686416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.835186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:17.835156 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.992053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:17.992026 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:18.155903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.155875 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:18.219576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.219514 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:41:18.258161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258132 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258227 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258303 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258505 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258607 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258572 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258607 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258510 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:18.258802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258622 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258766 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258832 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.258918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258902 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy\") pod \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\" (UID: \"01ffc4c1-3d5f-4be5-a663-e5aa035fce83\") " Oct 30 13:41:18.259026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258944 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:18.259026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258973 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:18.259182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.259143 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:18.259704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.259676 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.259809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.259706 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.259809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.259720 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.259809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.259736 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.259809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.258914 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:18.260742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.260716 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "kube-api-access-8bdzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:18.261029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.261005 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:18.261107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.261033 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:18.261172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.261156 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy" pod "01ffc4c1-3d5f-4be5-a663-e5aa035fce83" (UID: "01ffc4c1-3d5f-4be5-a663-e5aa035fce83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:18.362099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.362063 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.362099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.362099 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.362444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.362113 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.362444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.362126 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdzb\" (UniqueName: \"kubernetes.io/projected/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-kube-api-access-8bdzb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.362444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.362140 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ffc4c1-3d5f-4be5-a663-e5aa035fce83-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.706968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.706916 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:18.843932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.843899 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:18.880834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.880803 5693 generic.go:356] "Generic (PLEG): container finished" podID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerID="cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a" exitCode=0 Oct 30 13:41:18.881021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.880883 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" event={"ID":"01ffc4c1-3d5f-4be5-a663-e5aa035fce83","Type":"ContainerDied","Data":"cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a"} Oct 30 13:41:18.881021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.880944 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" Oct 30 13:41:18.881021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.880959 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4" event={"ID":"01ffc4c1-3d5f-4be5-a663-e5aa035fce83","Type":"ContainerDied","Data":"3d7f5e5e780f5f475ab2b54c0de21df05ea22950859c857df92b9cc7bd55ad19"} Oct 30 13:41:18.881021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.880984 5693 scope.go:122] "RemoveContainer" containerID="cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a" Oct 30 13:41:18.894922 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.894896 5693 scope.go:122] "RemoveContainer" containerID="cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a" Oct 30 13:41:18.895231 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:41:18.895205 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a\": container with ID starting with cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a not found: ID does not exist" containerID="cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a" Oct 30 13:41:18.895311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.895242 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a"} err="failed to get container status \"cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a\": rpc error: code = NotFound desc = could not find container \"cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a\": container with ID starting with cb8ff68682e4c4f1f05eaa9e2a3ea1efb764ac8eabcfba57821ed9215ed1107a not found: ID does not exist" Oct 30 13:41:18.958678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.952665 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4"] Oct 30 13:41:18.970624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:18.967234 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-thhj4"] Oct 30 13:41:19.892729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:19.892700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:20.130817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.130782 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb"] Oct 30 13:41:20.131735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.131718 5693 cpu_manager.go:414] "containerMap: removing container" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" Oct 30 13:41:20.131735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.131735 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" Oct 30 13:41:20.131908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.131896 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" containerName="istio-proxy" Oct 30 13:41:20.137315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.137297 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.158151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.158123 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb"] Oct 30 13:41:20.163510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.163484 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"agc-cred-a60789018bef\"" Oct 30 13:41:20.164633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.164500 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"openshift-service-ca.crt\"" Oct 30 13:41:20.174669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.170653 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a75d02\"" Oct 30 13:41:20.174956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.171267 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"payload-processing-external-model-e2e--08fbcc8d-dockercfg-724q4\"" Oct 30 13:41:20.175098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.171344 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"kube-root-ca.crt\"" Oct 30 13:41:20.175098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.171432 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"routing-overlay\"" Oct 30 13:41:20.318390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.318320 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblxz\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.318701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.318668 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.319048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.319019 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.319969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.319265 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.368681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.366939 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ffc4c1-3d5f-4be5-a663-e5aa035fce83" path="/var/lib/kubelet/pods/01ffc4c1-3d5f-4be5-a663-e5aa035fce83/volumes" Oct 30 13:41:20.421478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.421352 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblxz\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.421478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.421410 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.421815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.421644 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.421815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.421763 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.422323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.422294 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.422433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.422417 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.424272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.424254 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.439884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.439858 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblxz\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.452845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.452819 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.743042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.742938 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:20.761997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.761925 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb"] Oct 30 13:41:20.898845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.898811 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" event={"ID":"64e20eee-6850-452c-9745-3934030f7860","Type":"ContainerStarted","Data":"6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4"} Oct 30 13:41:20.899233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.898878 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" event={"ID":"64e20eee-6850-452c-9745-3934030f7860","Type":"ContainerStarted","Data":"3b45be90b73da78e8c7e27ede8bf416f869b06ca5d67794f314bc391adfa41c1"} Oct 30 13:41:20.899233 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.899007 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:20.937875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:20.937821 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" podStartSLOduration=0.937806651 podStartE2EDuration="937.806651ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:20.935072403 +0000 UTC m=+2256.801690576" watchObservedRunningTime="2026-10-30 13:41:20.937806651 +0000 UTC m=+2256.804424823" Oct 30 13:41:22.450067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:22.450003 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:23.013629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:23.013557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:23.101510 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:41:23.100375 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:41:23.168855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:23.168822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:24.270864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:24.270817 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:24.599384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:24.599305 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:24.820856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:24.820827 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:41:25.875914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:25.875879 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:27.531772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:27.531741 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:29.230310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:29.230264 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:30.760096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:30.760065 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:31.005778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:31.005746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:31.146434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:31.146343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:31.291418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:31.291384 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:31.911215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:31.911187 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:41:32.342548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:32.342502 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:33.913743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:33.913711 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:34.046812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:34.046769 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:35.508340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:35.508303 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:35.649869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:35.649838 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:35.794986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:35.794907 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:36.184253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:36.184169 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:37.081207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:37.081172 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:37.220301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:37.220269 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:37.740322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:37.740291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:38.169523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:38.169443 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:39.406876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:39.406842 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:41.242268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:41.242228 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:42.568921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:42.568880 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:42.716942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:42.716911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:42.882915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:42.882829 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:44.570795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:44.570760 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:44.726672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:44.726580 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:44.948314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:44.948236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:45.575466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:45.575409 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:45.776627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:45.754574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:46.398122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:46.398082 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:46.923048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:46.923016 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc"] Oct 30 13:41:46.928214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:46.928191 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:46.945997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:46.945966 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc"] Oct 30 13:41:46.953345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:46.953315 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-66971aa113bb\"" Oct 30 13:41:47.008938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.008875 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.009274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.008960 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmxq\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.009481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.009456 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.010023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.009664 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.072299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.072266 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88"] Oct 30 13:41:47.072491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.072471 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" containerID="cri-o://551a53ad7e69765bc326997897b7422568cfa9bcbad99aab11170b7cde27883b" gracePeriod=30 Oct 30 13:41:47.110968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.110934 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.111169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.111110 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.111169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.111151 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmxq\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.111568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.111539 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.111754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.111729 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.112264 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.112242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.114066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.114046 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.131023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.130988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmxq\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.243321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.243292 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:47.550523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:47.550495 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc"] Oct 30 13:41:47.552692 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:41:47.552665 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3bbc46_f8e1_4815_b44d_424e94afba99.slice/crio-4ea274d49653e072600a0af870c348e284e10e059d6ca4cc546c3f039a9b8a49 WatchSource:0}: Error finding container 4ea274d49653e072600a0af870c348e284e10e059d6ca4cc546c3f039a9b8a49: Status 404 returned error can't find the container with id 4ea274d49653e072600a0af870c348e284e10e059d6ca4cc546c3f039a9b8a49 Oct 30 13:41:48.099097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:48.099055 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:48.205276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:48.205238 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" event={"ID":"be3bbc46-f8e1-4815-b44d-424e94afba99","Type":"ContainerStarted","Data":"13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e"} Oct 30 13:41:48.205460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:48.205289 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" event={"ID":"be3bbc46-f8e1-4815-b44d-424e94afba99","Type":"ContainerStarted","Data":"4ea274d49653e072600a0af870c348e284e10e059d6ca4cc546c3f039a9b8a49"} Oct 30 13:41:48.205513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:48.205479 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:48.245343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:48.245290 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" podStartSLOduration=2.245276274 podStartE2EDuration="2.245276274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:48.241831638 +0000 UTC m=+2284.108449823" watchObservedRunningTime="2026-10-30 13:41:48.245276274 +0000 UTC m=+2284.111894446" Oct 30 13:41:49.446627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:49.446567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:49.590323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:49.590288 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:49.738624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:49.734795 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:49.959359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:49.959321 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:50.180448 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:50.180333 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:50.856913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:50.856874 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:51.847697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:51.847665 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:52.360404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.360329 5693 generic.go:356] "Generic (PLEG): container finished" podID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerID="551a53ad7e69765bc326997897b7422568cfa9bcbad99aab11170b7cde27883b" exitCode=0 Oct 30 13:41:52.360791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.360404 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" event={"ID":"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99","Type":"ContainerDied","Data":"551a53ad7e69765bc326997897b7422568cfa9bcbad99aab11170b7cde27883b"} Oct 30 13:41:52.438317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.438290 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:41:52.535779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.535740 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.535964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.535852 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.535964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.535905 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.535964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.535945 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.536163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.536163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536091 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.536279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536251 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:52.536339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536268 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:52.536763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536732 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:52.536909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536806 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:52.536909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536838 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.537040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.536967 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.538279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.537368 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket\") pod \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\" (UID: \"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99\") " Oct 30 13:41:52.538541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.538256 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:52.539383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.539358 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.539509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.539387 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.539509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.539400 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.539509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.539412 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.539509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.539426 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.545950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.545695 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:52.545950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.545764 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:52.545950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.545769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:52.545950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.545848 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25" pod "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" (UID: "a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99"). InnerVolumeSpecName "kube-api-access-4gw25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:52.641070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.640990 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.641070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.641022 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gw25\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-kube-api-access-4gw25\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.641070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.641033 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:52.641070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:52.641042 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.057630 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:41:53.057567 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:41:53.370844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:53.370758 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" Oct 30 13:41:53.370844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:53.370795 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88" event={"ID":"a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99","Type":"ContainerDied","Data":"02ef70a05801dfc41bd5a78822497f14abc10a7c83dce9af6686bba3789b3eac"} Oct 30 13:41:53.371338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:53.370903 5693 scope.go:122] "RemoveContainer" containerID="551a53ad7e69765bc326997897b7422568cfa9bcbad99aab11170b7cde27883b" Oct 30 13:41:53.434455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:53.434421 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88"] Oct 30 13:41:53.446069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:53.446039 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-7fq88"] Oct 30 13:41:53.471099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:53.471066 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:54.256139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:54.256105 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" path="/var/lib/kubelet/pods/a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99/volumes" Oct 30 13:41:54.776371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:54.776334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:54.935798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:54.935759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:55.123108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.123025 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2"] Oct 30 13:41:55.123934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.123915 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" Oct 30 13:41:55.123999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.123935 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" Oct 30 13:41:55.124071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.124047 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f94fbb-1bf6-4a48-8ac9-0fcdb78cfc99" containerName="istio-proxy" Oct 30 13:41:55.129245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.129223 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.148082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.148056 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2"] Oct 30 13:41:55.213697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.213661 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.213890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.213861 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnprh\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.213961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.213911 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.214051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.214024 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.316713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.316672 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnprh\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.316919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.316788 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.316919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.316866 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.317053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.316960 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.319182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.319151 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.320679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.320648 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.322750 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.322726 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:55.325188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.325158 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.379665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.373793 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnprh\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.445952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.445911 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:55.810732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.810703 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2"] Oct 30 13:41:55.812678 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:41:55.812644 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880cc58b_2d9d_4ac6_bc42_27fe457dc153.slice/crio-6513d3f9696d3924a49ea75c1942999f9cea111042d2bfd192b0d08f19dbed35 WatchSource:0}: Error finding container 6513d3f9696d3924a49ea75c1942999f9cea111042d2bfd192b0d08f19dbed35: Status 404 returned error can't find the container with id 6513d3f9696d3924a49ea75c1942999f9cea111042d2bfd192b0d08f19dbed35 Oct 30 13:41:55.828151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:55.828122 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:56.417831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:56.417792 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" event={"ID":"880cc58b-2d9d-4ac6-bc42-27fe457dc153","Type":"ContainerStarted","Data":"943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417"} Oct 30 13:41:56.418023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:56.417851 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" event={"ID":"880cc58b-2d9d-4ac6-bc42-27fe457dc153","Type":"ContainerStarted","Data":"6513d3f9696d3924a49ea75c1942999f9cea111042d2bfd192b0d08f19dbed35"} Oct 30 13:41:56.418023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:56.417877 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:41:56.461189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:56.461141 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" podStartSLOduration=1.461126266 podStartE2EDuration="1.461126266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:56.457850578 +0000 UTC m=+2292.324468773" watchObservedRunningTime="2026-10-30 13:41:56.461126266 +0000 UTC m=+2292.327744438" Oct 30 13:41:56.996671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:56.996555 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:58.639628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:58.623797 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:59.217930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:59.217902 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:41:59.309372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:41:59.309331 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq"] Oct 30 13:42:00.186924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:00.186849 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:01.818140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:01.818102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:03.589572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:03.588900 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:04.310185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.310129 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" podUID="06f8404f-93e6-4565-91dd-32ef601f33f6" containerName="payload-processing" containerID="cri-o://c94c944f1bb27de178d3841b6bddb234261468300944a2ab2bdeb339b946eee3" gracePeriod=25 Oct 30 13:42:04.481460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.481426 5693 generic.go:356] "Generic (PLEG): container finished" podID="06f8404f-93e6-4565-91dd-32ef601f33f6" containerID="c94c944f1bb27de178d3841b6bddb234261468300944a2ab2bdeb339b946eee3" exitCode=0 Oct 30 13:42:04.481676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.481502 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" event={"ID":"06f8404f-93e6-4565-91dd-32ef601f33f6","Type":"ContainerDied","Data":"c94c944f1bb27de178d3841b6bddb234261468300944a2ab2bdeb339b946eee3"} Oct 30 13:42:04.586723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.586694 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:42:04.679156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.679108 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9\") pod \"06f8404f-93e6-4565-91dd-32ef601f33f6\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " Oct 30 13:42:04.679694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.679199 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume\") pod \"06f8404f-93e6-4565-91dd-32ef601f33f6\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " Oct 30 13:42:04.679694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.679270 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials\") pod \"06f8404f-93e6-4565-91dd-32ef601f33f6\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " Oct 30 13:42:04.679694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.679430 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume\") pod \"06f8404f-93e6-4565-91dd-32ef601f33f6\" (UID: \"06f8404f-93e6-4565-91dd-32ef601f33f6\") " Oct 30 13:42:04.679867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.679668 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume" pod "06f8404f-93e6-4565-91dd-32ef601f33f6" (UID: "06f8404f-93e6-4565-91dd-32ef601f33f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:04.679867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.679835 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume" pod "06f8404f-93e6-4565-91dd-32ef601f33f6" (UID: "06f8404f-93e6-4565-91dd-32ef601f33f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:04.680258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.680236 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:04.680319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.680266 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f8404f-93e6-4565-91dd-32ef601f33f6-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:04.681708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.681685 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials" pod "06f8404f-93e6-4565-91dd-32ef601f33f6" (UID: "06f8404f-93e6-4565-91dd-32ef601f33f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:04.681809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.681789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9" pod "06f8404f-93e6-4565-91dd-32ef601f33f6" (UID: "06f8404f-93e6-4565-91dd-32ef601f33f6"). InnerVolumeSpecName "kube-api-access-d68m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:04.781773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.781733 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68m9\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-kube-api-access-d68m9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:04.781773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:04.781769 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f8404f-93e6-4565-91dd-32ef601f33f6-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:05.211828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:05.211781 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:05.490086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:05.490059 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" Oct 30 13:42:05.490086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:05.490076 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq" event={"ID":"06f8404f-93e6-4565-91dd-32ef601f33f6","Type":"ContainerDied","Data":"76e46f818bb056bfc03384f6f9bddb8617790a3431e796dc8f02f055f1f136d8"} Oct 30 13:42:05.490323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:05.490137 5693 scope.go:122] "RemoveContainer" containerID="c94c944f1bb27de178d3841b6bddb234261468300944a2ab2bdeb339b946eee3" Oct 30 13:42:05.541935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:05.541905 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq"] Oct 30 13:42:05.553348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:05.553322 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-579724xhq"] Oct 30 13:42:06.255952 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:06.255917 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f8404f-93e6-4565-91dd-32ef601f33f6" path="/var/lib/kubelet/pods/06f8404f-93e6-4565-91dd-32ef601f33f6/volumes" Oct 30 13:42:06.889484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:06.889457 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:07.039511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.039454 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:07.211350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.211267 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2"] Oct 30 13:42:07.214686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.214662 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:42:07.243983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.243947 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb"] Oct 30 13:42:07.244999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.244962 5693 cpu_manager.go:414] "containerMap: removing container" podUID="06f8404f-93e6-4565-91dd-32ef601f33f6" containerName="payload-processing" Oct 30 13:42:07.244999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.244983 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f8404f-93e6-4565-91dd-32ef601f33f6" containerName="payload-processing" Oct 30 13:42:07.245212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.245108 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f8404f-93e6-4565-91dd-32ef601f33f6" containerName="payload-processing" Oct 30 13:42:07.250655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.250624 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.265553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.265529 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb"] Oct 30 13:42:07.318487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.318450 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.318754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.318555 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nz8\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.318754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.318691 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.318917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.318893 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.420811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.420774 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.421025 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.420920 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.421101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.421030 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nz8\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.421164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.421106 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.421639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.421592 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.421947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.421928 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.423560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.423541 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.491618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.477378 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nz8\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.577632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.576877 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:07.905472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:07.905444 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb"] Oct 30 13:42:07.908168 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:42:07.908140 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f31d7f_2265_4b3f_9160_38a860e6f139.slice/crio-1d903b10796df19615e4773f69c33935f282a52223cb401baecb9e17e297c4b1 WatchSource:0}: Error finding container 1d903b10796df19615e4773f69c33935f282a52223cb401baecb9e17e297c4b1: Status 404 returned error can't find the container with id 1d903b10796df19615e4773f69c33935f282a52223cb401baecb9e17e297c4b1 Oct 30 13:42:08.545960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:08.545925 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" event={"ID":"15f31d7f-2265-4b3f-9160-38a860e6f139","Type":"ContainerStarted","Data":"5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180"} Oct 30 13:42:08.546434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:08.546004 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" event={"ID":"15f31d7f-2265-4b3f-9160-38a860e6f139","Type":"ContainerStarted","Data":"1d903b10796df19615e4773f69c33935f282a52223cb401baecb9e17e297c4b1"} Oct 30 13:42:08.546434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:08.546164 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:08.589545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:08.589495 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" podStartSLOduration=1.589482409 podStartE2EDuration="1.589482409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:08.58746141 +0000 UTC m=+2304.454079586" watchObservedRunningTime="2026-10-30 13:42:08.589482409 +0000 UTC m=+2304.456100582" Oct 30 13:42:08.695397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:08.695357 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:10.328578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:10.328542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:12.042688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.042646 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:12.212017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.211980 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" podUID="880cc58b-2d9d-4ac6-bc42-27fe457dc153" containerName="payload-processing" containerID="cri-o://943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417" gracePeriod=25 Oct 30 13:42:12.485069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.485043 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:42:12.578222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.578183 5693 generic.go:356] "Generic (PLEG): container finished" podID="880cc58b-2d9d-4ac6-bc42-27fe457dc153" containerID="943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417" exitCode=0 Oct 30 13:42:12.578419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.578231 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" Oct 30 13:42:12.578419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.578272 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" event={"ID":"880cc58b-2d9d-4ac6-bc42-27fe457dc153","Type":"ContainerDied","Data":"943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417"} Oct 30 13:42:12.578419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.578329 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2" event={"ID":"880cc58b-2d9d-4ac6-bc42-27fe457dc153","Type":"ContainerDied","Data":"6513d3f9696d3924a49ea75c1942999f9cea111042d2bfd192b0d08f19dbed35"} Oct 30 13:42:12.578419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.578356 5693 scope.go:122] "RemoveContainer" containerID="943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417" Oct 30 13:42:12.590300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.590283 5693 scope.go:122] "RemoveContainer" containerID="943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417" Oct 30 13:42:12.590645 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:12.590580 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417\": container with ID starting with 943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417 not found: ID does not exist" containerID="943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417" Oct 30 13:42:12.590767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.590646 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417"} err="failed to get container status \"943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417\": rpc error: code = NotFound desc = could not find container \"943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417\": container with ID starting with 943a909caf97445b563dff881bbae3f4acc0c00bfac6839856fe8e9481b41417 not found: ID does not exist" Oct 30 13:42:12.604163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.604137 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume\") pod \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " Oct 30 13:42:12.604289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.604185 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials\") pod \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " Oct 30 13:42:12.604436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.604422 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh\") pod \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " Oct 30 13:42:12.604942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.604548 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume\") pod \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\" (UID: \"880cc58b-2d9d-4ac6-bc42-27fe457dc153\") " Oct 30 13:42:12.604942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.604676 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume" pod "880cc58b-2d9d-4ac6-bc42-27fe457dc153" (UID: "880cc58b-2d9d-4ac6-bc42-27fe457dc153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:12.605074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.605025 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume" pod "880cc58b-2d9d-4ac6-bc42-27fe457dc153" (UID: "880cc58b-2d9d-4ac6-bc42-27fe457dc153"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:12.605956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.605357 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:12.605956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.605381 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880cc58b-2d9d-4ac6-bc42-27fe457dc153-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:12.608363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.608336 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials" pod "880cc58b-2d9d-4ac6-bc42-27fe457dc153" (UID: "880cc58b-2d9d-4ac6-bc42-27fe457dc153"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:12.608473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.608377 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh" pod "880cc58b-2d9d-4ac6-bc42-27fe457dc153" (UID: "880cc58b-2d9d-4ac6-bc42-27fe457dc153"). InnerVolumeSpecName "kube-api-access-rnprh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:12.706746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.706709 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:12.706746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.706738 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnprh\" (UniqueName: \"kubernetes.io/projected/880cc58b-2d9d-4ac6-bc42-27fe457dc153-kube-api-access-rnprh\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:12.925511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.925476 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2"] Oct 30 13:42:12.937110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:12.937076 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bsfqj2"] Oct 30 13:42:13.883195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:13.877382 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:14.256056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:14.256025 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880cc58b-2d9d-4ac6-bc42-27fe457dc153" path="/var/lib/kubelet/pods/880cc58b-2d9d-4ac6-bc42-27fe457dc153/volumes" Oct 30 13:42:15.529329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:15.529292 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:17.142624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:17.135923 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:17.379067 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.379036 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-6acca70650d1: secret "agc-cred-6acca70650d1" not found Oct 30 13:42:17.379302 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.379286 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-98ec1e033fce: secret "agc-cred-98ec1e033fce" not found Oct 30 13:42:17.379532 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.379507 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf: [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:17.379661 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.379616 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials podName:207fcd02-550e-4270-ad9b-654511d09553 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:17.879579509 +0000 UTC m=+2313.746197675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" (UID: "207fcd02-550e-4270-ad9b-654511d09553") : [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:17.886856 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.886822 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-6acca70650d1: secret "agc-cred-6acca70650d1" not found Oct 30 13:42:17.886856 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.886859 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-98ec1e033fce: secret "agc-cred-98ec1e033fce" not found Oct 30 13:42:17.887076 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.886869 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf: [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:17.887076 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:17.886923 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials podName:207fcd02-550e-4270-ad9b-654511d09553 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:18.886908716 +0000 UTC m=+2314.753526870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" (UID: "207fcd02-550e-4270-ad9b-654511d09553") : [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:18.788268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:18.788235 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:18.842408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:18.842377 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb"] Oct 30 13:42:18.845260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:18.845232 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:18.855169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:18.855132 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf"] Oct 30 13:42:18.907941 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:18.907906 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-6acca70650d1: secret "agc-cred-6acca70650d1" not found Oct 30 13:42:18.907941 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:18.907928 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-94a5564a5433: secret "agc-cred-94a5564a5433" not found Oct 30 13:42:18.907941 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:18.907934 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-98ec1e033fce: secret "agc-cred-98ec1e033fce" not found Oct 30 13:42:18.907941 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:18.907943 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf: [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-94a5564a5433" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:18.908290 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:18.907991 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials podName:207fcd02-550e-4270-ad9b-654511d09553 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:20.907977062 +0000 UTC m=+2316.774595213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" (UID: "207fcd02-550e-4270-ad9b-654511d09553") : [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-94a5564a5433" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:18.930675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:18.930641 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:19.064979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:19.064898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:19.531580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:19.531542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:20.443437 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:20.443396 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:20.968908 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:20.968876 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-6acca70650d1: secret "agc-cred-6acca70650d1" not found Oct 30 13:42:20.968908 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:20.968903 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-94a5564a5433: secret "agc-cred-94a5564a5433" not found Oct 30 13:42:20.968908 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:20.968910 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e3eb98/agc-cred-98ec1e033fce: secret "agc-cred-98ec1e033fce" not found Oct 30 13:42:20.969187 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:20.968920 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf: [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-94a5564a5433" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:20.969187 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:20.969002 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials podName:207fcd02-550e-4270-ad9b-654511d09553 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:24.968987064 +0000 UTC m=+2320.835605216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" (UID: "207fcd02-550e-4270-ad9b-654511d09553") : [secret "agc-cred-6acca70650d1" not found, secret "agc-cred-94a5564a5433" not found, secret "agc-cred-98ec1e033fce" not found] Oct 30 13:42:22.032154 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:22.032118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:23.057172 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:23.057132 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:42:23.571096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:23.570926 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:23.843630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:23.843544 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" podUID="15f31d7f-2265-4b3f-9160-38a860e6f139" containerName="payload-processing" containerID="cri-o://5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180" gracePeriod=25 Oct 30 13:42:23.855691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:23.855640 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" podUID="207fcd02-550e-4270-ad9b-654511d09553" containerName="payload-processing" containerID="cri-o://a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4" gracePeriod=25 Oct 30 13:42:24.153873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.153846 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:24.174414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.174386 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:42:24.324921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.324885 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume\") pod \"207fcd02-550e-4270-ad9b-654511d09553\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " Oct 30 13:42:24.325142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.324959 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials\") pod \"207fcd02-550e-4270-ad9b-654511d09553\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " Oct 30 13:42:24.325142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325021 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume\") pod \"207fcd02-550e-4270-ad9b-654511d09553\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " Oct 30 13:42:24.325267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325210 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8\") pod \"15f31d7f-2265-4b3f-9160-38a860e6f139\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " Oct 30 13:42:24.325330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325270 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume\") pod \"15f31d7f-2265-4b3f-9160-38a860e6f139\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " Oct 30 13:42:24.325388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325328 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22\") pod \"207fcd02-550e-4270-ad9b-654511d09553\" (UID: \"207fcd02-550e-4270-ad9b-654511d09553\") " Oct 30 13:42:24.325388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325380 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume\") pod \"15f31d7f-2265-4b3f-9160-38a860e6f139\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " Oct 30 13:42:24.325490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325400 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume" pod "207fcd02-550e-4270-ad9b-654511d09553" (UID: "207fcd02-550e-4270-ad9b-654511d09553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:24.325490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325415 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume" pod "207fcd02-550e-4270-ad9b-654511d09553" (UID: "207fcd02-550e-4270-ad9b-654511d09553"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:24.325637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325548 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials\") pod \"15f31d7f-2265-4b3f-9160-38a860e6f139\" (UID: \"15f31d7f-2265-4b3f-9160-38a860e6f139\") " Oct 30 13:42:24.325857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325827 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume" pod "15f31d7f-2265-4b3f-9160-38a860e6f139" (UID: "15f31d7f-2265-4b3f-9160-38a860e6f139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:24.326027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.325845 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume" pod "15f31d7f-2265-4b3f-9160-38a860e6f139" (UID: "15f31d7f-2265-4b3f-9160-38a860e6f139"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:24.326344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.326315 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.326458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.326348 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.326458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.326364 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/207fcd02-550e-4270-ad9b-654511d09553-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.326458 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.326380 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f31d7f-2265-4b3f-9160-38a860e6f139-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.327631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.327569 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8" pod "15f31d7f-2265-4b3f-9160-38a860e6f139" (UID: "15f31d7f-2265-4b3f-9160-38a860e6f139"). InnerVolumeSpecName "kube-api-access-j9nz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:24.327804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.327779 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials" pod "15f31d7f-2265-4b3f-9160-38a860e6f139" (UID: "15f31d7f-2265-4b3f-9160-38a860e6f139"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:24.327804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.327791 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22" pod "207fcd02-550e-4270-ad9b-654511d09553" (UID: "207fcd02-550e-4270-ad9b-654511d09553"). InnerVolumeSpecName "kube-api-access-vvq22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:24.327898 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.327883 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials" pod "207fcd02-550e-4270-ad9b-654511d09553" (UID: "207fcd02-550e-4270-ad9b-654511d09553"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:24.428423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.428335 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.428585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.428444 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.428585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.428461 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nz8\" (UniqueName: \"kubernetes.io/projected/15f31d7f-2265-4b3f-9160-38a860e6f139-kube-api-access-j9nz8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.428585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.428473 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq22\" (UniqueName: \"kubernetes.io/projected/207fcd02-550e-4270-ad9b-654511d09553-kube-api-access-vvq22\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:24.679586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.679492 5693 generic.go:356] "Generic (PLEG): container finished" podID="15f31d7f-2265-4b3f-9160-38a860e6f139" containerID="5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180" exitCode=0 Oct 30 13:42:24.679586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.679543 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" Oct 30 13:42:24.679847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.679577 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" event={"ID":"15f31d7f-2265-4b3f-9160-38a860e6f139","Type":"ContainerDied","Data":"5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180"} Oct 30 13:42:24.679847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.679647 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb" event={"ID":"15f31d7f-2265-4b3f-9160-38a860e6f139","Type":"ContainerDied","Data":"1d903b10796df19615e4773f69c33935f282a52223cb401baecb9e17e297c4b1"} Oct 30 13:42:24.679847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.679673 5693 scope.go:122] "RemoveContainer" containerID="5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180" Oct 30 13:42:24.681405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.681378 5693 generic.go:356] "Generic (PLEG): container finished" podID="207fcd02-550e-4270-ad9b-654511d09553" containerID="a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4" exitCode=0 Oct 30 13:42:24.681544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.681418 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" Oct 30 13:42:24.681544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.681500 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" event={"ID":"207fcd02-550e-4270-ad9b-654511d09553","Type":"ContainerDied","Data":"a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4"} Oct 30 13:42:24.681544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.681534 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf" event={"ID":"207fcd02-550e-4270-ad9b-654511d09553","Type":"ContainerDied","Data":"3ccfab12f524ad660ae09ee718031469b84e04c7047b6b2fb6f90b1be9e4b08b"} Oct 30 13:42:24.690863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.690838 5693 scope.go:122] "RemoveContainer" containerID="5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180" Oct 30 13:42:24.691204 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:24.691180 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180\": container with ID starting with 5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180 not found: ID does not exist" containerID="5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180" Oct 30 13:42:24.691315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.691216 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180"} err="failed to get container status \"5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180\": rpc error: code = NotFound desc = could not find container \"5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180\": container with ID starting with 5f71134788acf0d5a48984d1828aad3535b0375f2bd1efb28828cb2fafd55180 not found: ID does not exist" Oct 30 13:42:24.691315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.691240 5693 scope.go:122] "RemoveContainer" containerID="a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4" Oct 30 13:42:24.701862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.701843 5693 scope.go:122] "RemoveContainer" containerID="a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4" Oct 30 13:42:24.702137 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:24.702116 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4\": container with ID starting with a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4 not found: ID does not exist" containerID="a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4" Oct 30 13:42:24.702215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.702148 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4"} err="failed to get container status \"a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4\": rpc error: code = NotFound desc = could not find container \"a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4\": container with ID starting with a2d50d1fc156fadb7b83c7a6f78a20a9d073f3e3915b7ad0f6627ccf5deedde4 not found: ID does not exist" Oct 30 13:42:24.731834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.731802 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf"] Oct 30 13:42:24.743111 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.743086 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-74b874qgf"] Oct 30 13:42:24.768055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.768024 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb"] Oct 30 13:42:24.771573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:24.771546 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f657vsxb"] Oct 30 13:42:25.220944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:25.220911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:25.992666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:25.992626 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:26.145631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:26.145586 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:26.255473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:26.255380 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f31d7f-2265-4b3f-9160-38a860e6f139" path="/var/lib/kubelet/pods/15f31d7f-2265-4b3f-9160-38a860e6f139/volumes" Oct 30 13:42:26.256089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:26.256066 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207fcd02-550e-4270-ad9b-654511d09553" path="/var/lib/kubelet/pods/207fcd02-550e-4270-ad9b-654511d09553/volumes" Oct 30 13:42:26.295451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:26.295410 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:27.013478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:27.013441 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:28.630105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:28.630070 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:30.226286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:30.226250 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:31.900029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:31.899994 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:32.729438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:32.729398 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:33.247619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:33.243715 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:33.402253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:33.402220 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:33.720090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:33.720015 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:33.860715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:33.860664 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:34.040295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.040246 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:34.573842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.573812 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:34.805547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.805509 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb"] Oct 30 13:42:34.806320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806302 5693 cpu_manager.go:414] "containerMap: removing container" podUID="207fcd02-550e-4270-ad9b-654511d09553" containerName="payload-processing" Oct 30 13:42:34.806389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806322 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207fcd02-550e-4270-ad9b-654511d09553" containerName="payload-processing" Oct 30 13:42:34.806389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806344 5693 cpu_manager.go:414] "containerMap: removing container" podUID="880cc58b-2d9d-4ac6-bc42-27fe457dc153" containerName="payload-processing" Oct 30 13:42:34.806389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806349 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880cc58b-2d9d-4ac6-bc42-27fe457dc153" containerName="payload-processing" Oct 30 13:42:34.806389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806375 5693 cpu_manager.go:414] "containerMap: removing container" podUID="15f31d7f-2265-4b3f-9160-38a860e6f139" containerName="payload-processing" Oct 30 13:42:34.806389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806381 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f31d7f-2265-4b3f-9160-38a860e6f139" containerName="payload-processing" Oct 30 13:42:34.806577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806485 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="207fcd02-550e-4270-ad9b-654511d09553" containerName="payload-processing" Oct 30 13:42:34.806577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806495 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="880cc58b-2d9d-4ac6-bc42-27fe457dc153" containerName="payload-processing" Oct 30 13:42:34.806577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.806506 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f31d7f-2265-4b3f-9160-38a860e6f139" containerName="payload-processing" Oct 30 13:42:34.811839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.811820 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:34.830718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.830659 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb"] Oct 30 13:42:34.845061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.845030 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"kube-root-ca.crt\"" Oct 30 13:42:34.845872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.845392 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-e2e--175bf124-dockercfg-zl98l\"" Oct 30 13:42:34.846112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.845473 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-plugins-e2e-extmod-a-310546\"" Oct 30 13:42:34.846236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.845530 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"openshift-service-ca.crt\"" Oct 30 13:42:34.846329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.845581 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"routing-overlay\"" Oct 30 13:42:34.846329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:34.845711 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"e2e-cmp-legacy-bf9d40-provider-cred\"" Oct 30 13:42:35.105634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.105506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.105634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.105577 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.105869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.105760 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.105869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.105826 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.207368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.207326 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.207572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.207407 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.207572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.207538 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.207785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.207622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.208159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.208126 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.208266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.208184 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.210244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.210221 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.226594 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.226572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4\") pod \"payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.430433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.430354 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:35.623408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.622553 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:35.736653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.736625 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb"] Oct 30 13:42:35.738448 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:42:35.738417 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d1f9d5_b522_4ed6_a28f_0103369c174b.slice/crio-6b2847d82db33607c0fac8a775dfb8e0bd8c8e8d6bb539141aaf20bda9f20019 WatchSource:0}: Error finding container 6b2847d82db33607c0fac8a775dfb8e0bd8c8e8d6bb539141aaf20bda9f20019: Status 404 returned error can't find the container with id 6b2847d82db33607c0fac8a775dfb8e0bd8c8e8d6bb539141aaf20bda9f20019 Oct 30 13:42:35.769412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:35.769376 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" event={"ID":"77d1f9d5-b522-4ed6-a28f-0103369c174b","Type":"ContainerStarted","Data":"6b2847d82db33607c0fac8a775dfb8e0bd8c8e8d6bb539141aaf20bda9f20019"} Oct 30 13:42:36.507024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:36.506971 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.653584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:36.653551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.812722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:36.812447 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" event={"ID":"77d1f9d5-b522-4ed6-a28f-0103369c174b","Type":"ContainerStarted","Data":"0bb27b9be0486efef217fbfddf91fc731bef2614dcfdfae5f20b339417cb548f"} Oct 30 13:42:36.813936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:36.813911 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:36.901396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:36.901333 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" podStartSLOduration=2.9013152829999997 podStartE2EDuration="2.901315283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:36.875015329 +0000 UTC m=+2332.741633503" watchObservedRunningTime="2026-10-30 13:42:36.901315283 +0000 UTC m=+2332.767933456" Oct 30 13:42:37.412270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.412233 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:37.692580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.692501 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9"] Oct 30 13:42:37.697447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.697426 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.715817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.715789 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9"] Oct 30 13:42:37.722944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.722917 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\"" Oct 30 13:42:37.722944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.722917 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\"" Oct 30 13:42:37.723172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.723088 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"e2e-cfl-e402e5-provider-cred\"" Oct 30 13:42:37.725099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.725063 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\"" Oct 30 13:42:37.725099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.725090 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-wk62z\"" Oct 30 13:42:37.725386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.725360 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\"" Oct 30 13:42:37.726473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.726443 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-b691dcc50cd7\"" Oct 30 13:42:37.760925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.760887 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pxr\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.761073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.761037 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.761142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.761076 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.761252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.761234 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.862177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.862136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pxr\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.862364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.862294 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.862364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.862348 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.862486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.862422 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.863253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.863220 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.863378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.863291 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.864946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.864921 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.881169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.881138 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pxr\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:37.944307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:37.944224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:38.013723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.013681 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:38.082521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.082491 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:38.429109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.429067 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9"] Oct 30 13:42:38.429733 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:42:38.429687 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee2d183_6964_4565_ba84_3fad2f938fd7.slice/crio-d765d6ba1eb0c288f6a03e60ab8bb71f29b200e6c6147054d82c1c874715edad WatchSource:0}: Error finding container d765d6ba1eb0c288f6a03e60ab8bb71f29b200e6c6147054d82c1c874715edad: Status 404 returned error can't find the container with id d765d6ba1eb0c288f6a03e60ab8bb71f29b200e6c6147054d82c1c874715edad Oct 30 13:42:38.832131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.832088 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" event={"ID":"3ee2d183-6964-4565-ba84-3fad2f938fd7","Type":"ContainerStarted","Data":"4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3"} Oct 30 13:42:38.832653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.832156 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:38.832653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.832181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" event={"ID":"3ee2d183-6964-4565-ba84-3fad2f938fd7","Type":"ContainerStarted","Data":"d765d6ba1eb0c288f6a03e60ab8bb71f29b200e6c6147054d82c1c874715edad"} Oct 30 13:42:38.871059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:38.871006 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" podStartSLOduration=1.8709760800000002 podStartE2EDuration="1.87097608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:38.869661813 +0000 UTC m=+2334.736279983" watchObservedRunningTime="2026-10-30 13:42:38.87097608 +0000 UTC m=+2334.737594254" Oct 30 13:42:39.375623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:39.374413 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:40.475206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:40.475167 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc"] Oct 30 13:42:40.475822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:40.475415 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" containerID="cri-o://874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e" gracePeriod=30 Oct 30 13:42:41.627360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.627324 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz"] Oct 30 13:42:41.631861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.631837 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.639992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.639963 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.640148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.640128 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.640202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.640185 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mf9\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.640257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.640221 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.649679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.649647 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz"] Oct 30 13:42:41.657247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.657221 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"agc-cred-08e42d570f47\"" Oct 30 13:42:41.657429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.657221 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c3efb2\"" Oct 30 13:42:41.657530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.657290 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"payload-processing-external-model-e2e--2f6d7ae8-dockercfg-5jh47\"" Oct 30 13:42:41.659926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.659893 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"openshift-service-ca.crt\"" Oct 30 13:42:41.660049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.659952 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"kube-root-ca.crt\"" Oct 30 13:42:41.660049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.659963 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"routing-overlay\"" Oct 30 13:42:41.743246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.743204 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.743860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.743830 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.744146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.744125 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mf9\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.744268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.744253 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.745962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.745905 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.746554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.746508 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.794619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.775713 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.801159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.778116 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mf9\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:41.948847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:41.948764 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:42.137569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.137532 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv"] Oct 30 13:42:42.142908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.142884 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.150986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.150956 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.151118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.151000 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.151242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.151220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdws\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.151367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.151351 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.157799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.157774 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"agc-cred-9340b4ba0767\"" Oct 30 13:42:42.163435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.163404 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv"] Oct 30 13:42:42.252649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.252593 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdws\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.253115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.253073 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.253225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.253165 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.253225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.253214 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.254708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.254681 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.254826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.254782 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.256382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.256356 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.257293 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:42:42.257270 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eee141_e935_4d93_ae97_6bf6222acde6.slice/crio-911dd5daacc42e1c21dca6a4dd3d00526cc065f156c390bd7009ff9b792175f4 WatchSource:0}: Error finding container 911dd5daacc42e1c21dca6a4dd3d00526cc065f156c390bd7009ff9b792175f4: Status 404 returned error can't find the container with id 911dd5daacc42e1c21dca6a4dd3d00526cc065f156c390bd7009ff9b792175f4 Oct 30 13:42:42.259102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.259081 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz"] Oct 30 13:42:42.272532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.272508 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdws\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.460652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.460618 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:42.760371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.760344 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:42.770807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.770777 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv"] Oct 30 13:42:42.772888 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:42:42.772862 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3530ff_502c_4919_83b1_cc5327bc6083.slice/crio-7cb04126881dfc3bef01e3c831b486550fc85a43a3df8a824fdab466f0709f00 WatchSource:0}: Error finding container 7cb04126881dfc3bef01e3c831b486550fc85a43a3df8a824fdab466f0709f00: Status 404 returned error can't find the container with id 7cb04126881dfc3bef01e3c831b486550fc85a43a3df8a824fdab466f0709f00 Oct 30 13:42:42.874588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.874556 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" event={"ID":"dc3530ff-502c-4919-83b1-cc5327bc6083","Type":"ContainerStarted","Data":"e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4"} Oct 30 13:42:42.874771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.874655 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" event={"ID":"dc3530ff-502c-4919-83b1-cc5327bc6083","Type":"ContainerStarted","Data":"7cb04126881dfc3bef01e3c831b486550fc85a43a3df8a824fdab466f0709f00"} Oct 30 13:42:42.876857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.876833 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" event={"ID":"b4eee141-e935-4d93-ae97-6bf6222acde6","Type":"ContainerStarted","Data":"322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1"} Oct 30 13:42:42.876968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.876878 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" event={"ID":"b4eee141-e935-4d93-ae97-6bf6222acde6","Type":"ContainerStarted","Data":"911dd5daacc42e1c21dca6a4dd3d00526cc065f156c390bd7009ff9b792175f4"} Oct 30 13:42:42.876968 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.876939 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:42.913551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.913498 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" podStartSLOduration=0.913483771 podStartE2EDuration="913.483771ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:42.912372782 +0000 UTC m=+2338.778990956" watchObservedRunningTime="2026-10-30 13:42:42.913483771 +0000 UTC m=+2338.780101944" Oct 30 13:42:42.956648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.954771 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" podStartSLOduration=1.954751865 podStartE2EDuration="1.954751865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:42.950870888 +0000 UTC m=+2338.817489080" watchObservedRunningTime="2026-10-30 13:42:42.954751865 +0000 UTC m=+2338.821370041" Oct 30 13:42:42.981647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:42.976355 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:43.885509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:43.885478 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:44.633516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:44.633476 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:44.787468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:44.787426 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:45.020745 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.020705 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a75d02/agc-cred-9340b4ba0767: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:45.020745 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.020745 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:45.021266 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.020817 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials podName:dc3530ff-502c-4919-83b1-cc5327bc6083 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:45.520795859 +0000 UTC m=+2341.387414009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083") : secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:45.041019 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.040986 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv"] Oct 30 13:42:45.548063 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.548030 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a75d02/agc-cred-9340b4ba0767: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:45.548063 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.548064 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:45.548331 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.548143 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials podName:dc3530ff-502c-4919-83b1-cc5327bc6083 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:46.548123137 +0000 UTC m=+2342.414741303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083") : secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:45.770387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.770324 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:42:45.853536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853498 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.853747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853637 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.853747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853685 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.853747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853732 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.854045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853823 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.854045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853899 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.854045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.853957 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.854045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.854043 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.854281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.854163 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5\") pod \"e47eff72-66ab-4409-9bd0-f801817d1a26\" (UID: \"e47eff72-66ab-4409-9bd0-f801817d1a26\") " Oct 30 13:42:45.854436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.854412 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:45.854518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.854453 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.854902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.854765 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.855074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.854974 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.855074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.855056 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.855205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.855084 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47eff72-66ab-4409-9bd0-f801817d1a26-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.855205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.855098 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.855434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.855404 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.857951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.857919 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.858411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.858380 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:45.858411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.859043 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "kube-api-access-kkcz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:45.859539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.859490 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token" pod "e47eff72-66ab-4409-9bd0-f801817d1a26" (UID: "e47eff72-66ab-4409-9bd0-f801817d1a26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:45.905251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.905198 5693 generic.go:356] "Generic (PLEG): container finished" podID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerID="874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e" exitCode=0 Oct 30 13:42:45.905462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.905254 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" Oct 30 13:42:45.905648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.905573 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" event={"ID":"e47eff72-66ab-4409-9bd0-f801817d1a26","Type":"ContainerDied","Data":"874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e"} Oct 30 13:42:45.905811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.905684 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc" event={"ID":"e47eff72-66ab-4409-9bd0-f801817d1a26","Type":"ContainerDied","Data":"dc6e0d849266a4da286ce2d3ab7628bf8eff65efbed797c40e7cd74074fb121f"} Oct 30 13:42:45.905811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.905720 5693 scope.go:122] "RemoveContainer" containerID="874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e" Oct 30 13:42:45.918478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.918460 5693 scope.go:122] "RemoveContainer" containerID="874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e" Oct 30 13:42:45.918822 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:45.918792 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e\": container with ID starting with 874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e not found: ID does not exist" containerID="874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e" Oct 30 13:42:45.918897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.918833 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e"} err="failed to get container status \"874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e\": rpc error: code = NotFound desc = could not find container \"874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e\": container with ID starting with 874d4bf97e71425adc16acd31c1566f18b805b01d289e580273eddf3b9b32f1e not found: ID does not exist" Oct 30 13:42:45.956736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.956614 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.956736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.956652 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.956736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.956668 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.956736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.956680 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.956736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.956694 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkcz5\" (UniqueName: \"kubernetes.io/projected/e47eff72-66ab-4409-9bd0-f801817d1a26-kube-api-access-kkcz5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.956736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.956712 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47eff72-66ab-4409-9bd0-f801817d1a26-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.960188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.960154 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc"] Oct 30 13:42:45.964865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:45.964831 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-9c2wc"] Oct 30 13:42:46.260869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:46.260831 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" path="/var/lib/kubelet/pods/e47eff72-66ab-4409-9bd0-f801817d1a26/volumes" Oct 30 13:42:46.283840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:46.283805 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz"] Oct 30 13:42:46.394727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:46.394694 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:46.528977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:46.528891 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:46.587621 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:46.587266 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a75d02/agc-cred-9340b4ba0767: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:46.587621 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:46.587314 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:46.587621 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:46.587383 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials podName:dc3530ff-502c-4919-83b1-cc5327bc6083 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:48.587362882 +0000 UTC m=+2344.453981050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083") : secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:46.970047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:46.969970 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.137516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:47.137485 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.843816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:47.843774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.978768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:47.978732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:48.280270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:48.280240 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:48.634233 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:48.634155 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a75d02/agc-cred-9340b4ba0767: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:48.634233 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:48.634187 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv: secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:48.634414 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:48.634243 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials podName:dc3530ff-502c-4919-83b1-cc5327bc6083 nodeName:}" failed. No retries permitted until 2026-10-30 13:42:52.634228986 +0000 UTC m=+2348.500847137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083") : secret "agc-cred-9340b4ba0767" not found Oct 30 13:42:48.835746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:48.835717 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:42:49.842948 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:49.842915 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:42:50.082853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:50.082798 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:50.906257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:50.906214 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" podUID="dc3530ff-502c-4919-83b1-cc5327bc6083" containerName="payload-processing" containerID="cri-o://e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4" gracePeriod=25 Oct 30 13:42:51.199674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.199646 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:51.285160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.285104 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" podUID="b4eee141-e935-4d93-ae97-6bf6222acde6" containerName="payload-processing" containerID="cri-o://322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1" gracePeriod=25 Oct 30 13:42:51.287007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.286977 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume\") pod \"dc3530ff-502c-4919-83b1-cc5327bc6083\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " Oct 30 13:42:51.287132 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.287078 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials\") pod \"dc3530ff-502c-4919-83b1-cc5327bc6083\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " Oct 30 13:42:51.287214 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.287147 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume\") pod \"dc3530ff-502c-4919-83b1-cc5327bc6083\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " Oct 30 13:42:51.287451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.287410 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume" pod "dc3530ff-502c-4919-83b1-cc5327bc6083" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:51.287542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.287426 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws\") pod \"dc3530ff-502c-4919-83b1-cc5327bc6083\" (UID: \"dc3530ff-502c-4919-83b1-cc5327bc6083\") " Oct 30 13:42:51.287885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.287849 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume" pod "dc3530ff-502c-4919-83b1-cc5327bc6083" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:51.288857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.288584 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.288857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.288640 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc3530ff-502c-4919-83b1-cc5327bc6083-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.291157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.290067 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials" pod "dc3530ff-502c-4919-83b1-cc5327bc6083" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:51.291157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.290155 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws" pod "dc3530ff-502c-4919-83b1-cc5327bc6083" (UID: "dc3530ff-502c-4919-83b1-cc5327bc6083"). InnerVolumeSpecName "kube-api-access-mxdws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:51.390701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.390665 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdws\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-kube-api-access-mxdws\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.390701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.390702 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc3530ff-502c-4919-83b1-cc5327bc6083-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.537647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.537588 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:51.592936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.592901 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume\") pod \"b4eee141-e935-4d93-ae97-6bf6222acde6\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " Oct 30 13:42:51.593144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.593008 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume\") pod \"b4eee141-e935-4d93-ae97-6bf6222acde6\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " Oct 30 13:42:51.593217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.593189 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials\") pod \"b4eee141-e935-4d93-ae97-6bf6222acde6\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " Oct 30 13:42:51.593340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.593319 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9\") pod \"b4eee141-e935-4d93-ae97-6bf6222acde6\" (UID: \"b4eee141-e935-4d93-ae97-6bf6222acde6\") " Oct 30 13:42:51.593802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.593774 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume" pod "b4eee141-e935-4d93-ae97-6bf6222acde6" (UID: "b4eee141-e935-4d93-ae97-6bf6222acde6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:51.594349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.594303 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume" pod "b4eee141-e935-4d93-ae97-6bf6222acde6" (UID: "b4eee141-e935-4d93-ae97-6bf6222acde6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:51.596921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.596887 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9" pod "b4eee141-e935-4d93-ae97-6bf6222acde6" (UID: "b4eee141-e935-4d93-ae97-6bf6222acde6"). InnerVolumeSpecName "kube-api-access-z4mf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:51.599890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.599851 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials" pod "b4eee141-e935-4d93-ae97-6bf6222acde6" (UID: "b4eee141-e935-4d93-ae97-6bf6222acde6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:51.695063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.695029 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.695063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.695057 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mf9\" (UniqueName: \"kubernetes.io/projected/b4eee141-e935-4d93-ae97-6bf6222acde6-kube-api-access-z4mf9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.695063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.695067 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.695285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.695078 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eee141-e935-4d93-ae97-6bf6222acde6-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:42:51.907183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.907144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:51.966541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.966510 5693 generic.go:356] "Generic (PLEG): container finished" podID="dc3530ff-502c-4919-83b1-cc5327bc6083" containerID="e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4" exitCode=0 Oct 30 13:42:51.966732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.966553 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" Oct 30 13:42:51.966732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.966584 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" event={"ID":"dc3530ff-502c-4919-83b1-cc5327bc6083","Type":"ContainerDied","Data":"e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4"} Oct 30 13:42:51.966732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.966658 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv" event={"ID":"dc3530ff-502c-4919-83b1-cc5327bc6083","Type":"ContainerDied","Data":"7cb04126881dfc3bef01e3c831b486550fc85a43a3df8a824fdab466f0709f00"} Oct 30 13:42:51.966732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.966681 5693 scope.go:122] "RemoveContainer" containerID="e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4" Oct 30 13:42:51.968329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.968302 5693 generic.go:356] "Generic (PLEG): container finished" podID="b4eee141-e935-4d93-ae97-6bf6222acde6" containerID="322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1" exitCode=0 Oct 30 13:42:51.968516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.968345 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" Oct 30 13:42:51.968516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.968396 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" event={"ID":"b4eee141-e935-4d93-ae97-6bf6222acde6","Type":"ContainerDied","Data":"322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1"} Oct 30 13:42:51.968516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.968433 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz" event={"ID":"b4eee141-e935-4d93-ae97-6bf6222acde6","Type":"ContainerDied","Data":"911dd5daacc42e1c21dca6a4dd3d00526cc065f156c390bd7009ff9b792175f4"} Oct 30 13:42:51.978247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.978228 5693 scope.go:122] "RemoveContainer" containerID="e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4" Oct 30 13:42:51.978581 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:51.978551 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4\": container with ID starting with e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4 not found: ID does not exist" containerID="e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4" Oct 30 13:42:51.978714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.978593 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4"} err="failed to get container status \"e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4\": rpc error: code = NotFound desc = could not find container \"e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4\": container with ID starting with e97170f7a0118fbda881cc84cd4a0ff49883b40598043f8af42cf8ca9dd0e7d4 not found: ID does not exist" Oct 30 13:42:51.978714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.978694 5693 scope.go:122] "RemoveContainer" containerID="322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1" Oct 30 13:42:51.987881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.987866 5693 scope.go:122] "RemoveContainer" containerID="322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1" Oct 30 13:42:51.988133 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:51.988115 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1\": container with ID starting with 322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1 not found: ID does not exist" containerID="322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1" Oct 30 13:42:51.988198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:51.988142 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1"} err="failed to get container status \"322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1\": rpc error: code = NotFound desc = could not find container \"322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1\": container with ID starting with 322a08544a4f14899985004ccdee2b840ef2e3b1a80dadef16e76dc92cd12bb1 not found: ID does not exist" Oct 30 13:42:52.020392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:52.020350 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv"] Oct 30 13:42:52.025923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:52.025895 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-54d7g7phv"] Oct 30 13:42:52.060094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:52.060066 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz"] Oct 30 13:42:52.063379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:52.063354 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-7bcfxtsxz"] Oct 30 13:42:52.257195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:52.257165 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4eee141-e935-4d93-ae97-6bf6222acde6" path="/var/lib/kubelet/pods/b4eee141-e935-4d93-ae97-6bf6222acde6/volumes" Oct 30 13:42:52.257678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:52.257654 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3530ff-502c-4919-83b1-cc5327bc6083" path="/var/lib/kubelet/pods/dc3530ff-502c-4919-83b1-cc5327bc6083/volumes" Oct 30 13:42:53.057462 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:42:53.057422 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:42:53.518736 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:53.518700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:55.214443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:55.214349 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:55.936707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:55.936673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:56.857650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:56.857584 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:58.577411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:42:58.577378 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:00.220621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:00.219003 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:00.734994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:00.734954 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb"] Oct 30 13:43:00.856196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:00.856165 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:01.005245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:01.005145 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:01.426113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:01.426023 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:02.150291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:02.150226 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:03.838447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:03.838411 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:05.473115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:05.473080 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:05.736182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:05.736142 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" podUID="64e20eee-6850-452c-9745-3934030f7860" containerName="payload-processing" containerID="cri-o://6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4" gracePeriod=25 Oct 30 13:43:06.019832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.019805 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:43:06.060698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.060658 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume\") pod \"64e20eee-6850-452c-9745-3934030f7860\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " Oct 30 13:43:06.060889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.060874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials\") pod \"64e20eee-6850-452c-9745-3934030f7860\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " Oct 30 13:43:06.060977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.060907 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume\") pod \"64e20eee-6850-452c-9745-3934030f7860\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " Oct 30 13:43:06.061041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.061027 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz\") pod \"64e20eee-6850-452c-9745-3934030f7860\" (UID: \"64e20eee-6850-452c-9745-3934030f7860\") " Oct 30 13:43:06.061101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.061047 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume" pod "64e20eee-6850-452c-9745-3934030f7860" (UID: "64e20eee-6850-452c-9745-3934030f7860"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:06.062479 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.062438 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume" pod "64e20eee-6850-452c-9745-3934030f7860" (UID: "64e20eee-6850-452c-9745-3934030f7860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:06.063124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.063074 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.063124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.063103 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64e20eee-6850-452c-9745-3934030f7860-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.066808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.066778 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz" pod "64e20eee-6850-452c-9745-3934030f7860" (UID: "64e20eee-6850-452c-9745-3934030f7860"). InnerVolumeSpecName "kube-api-access-mblxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:06.069105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.069080 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials" pod "64e20eee-6850-452c-9745-3934030f7860" (UID: "64e20eee-6850-452c-9745-3934030f7860"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:06.076802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.076775 5693 generic.go:356] "Generic (PLEG): container finished" podID="64e20eee-6850-452c-9745-3934030f7860" containerID="6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4" exitCode=0 Oct 30 13:43:06.076929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.076843 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" Oct 30 13:43:06.076929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.076865 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" event={"ID":"64e20eee-6850-452c-9745-3934030f7860","Type":"ContainerDied","Data":"6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4"} Oct 30 13:43:06.077039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.076938 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb" event={"ID":"64e20eee-6850-452c-9745-3934030f7860","Type":"ContainerDied","Data":"3b45be90b73da78e8c7e27ede8bf416f869b06ca5d67794f314bc391adfa41c1"} Oct 30 13:43:06.077039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.076972 5693 scope.go:122] "RemoveContainer" containerID="6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4" Oct 30 13:43:06.100345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.100322 5693 scope.go:122] "RemoveContainer" containerID="6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4" Oct 30 13:43:06.100761 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:43:06.100733 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4\": container with ID starting with 6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4 not found: ID does not exist" containerID="6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4" Oct 30 13:43:06.100837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.100774 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4"} err="failed to get container status \"6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4\": rpc error: code = NotFound desc = could not find container \"6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4\": container with ID starting with 6931b493628222e7165f26f4b74d710ccd8b288b49e45f6f2ef77ba909a861a4 not found: ID does not exist" Oct 30 13:43:06.132185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.132156 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb"] Oct 30 13:43:06.135628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.135569 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-5998mvtwb"] Oct 30 13:43:06.165024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.164996 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.165024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.165023 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblxz\" (UniqueName: \"kubernetes.io/projected/64e20eee-6850-452c-9745-3934030f7860-kube-api-access-mblxz\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.256282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:06.256193 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e20eee-6850-452c-9745-3934030f7860" path="/var/lib/kubelet/pods/64e20eee-6850-452c-9745-3934030f7860/volumes" Oct 30 13:43:07.108530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:07.108501 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:07.340303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:07.340271 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:07.518624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:07.511103 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:08.406257 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.406212 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:08.530678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.530643 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp"] Oct 30 13:43:08.531698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531654 5693 cpu_manager.go:414] "containerMap: removing container" podUID="64e20eee-6850-452c-9745-3934030f7860" containerName="payload-processing" Oct 30 13:43:08.531698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531674 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e20eee-6850-452c-9745-3934030f7860" containerName="payload-processing" Oct 30 13:43:08.531698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531699 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" Oct 30 13:43:08.531918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531706 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" Oct 30 13:43:08.531918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531733 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b4eee141-e935-4d93-ae97-6bf6222acde6" containerName="payload-processing" Oct 30 13:43:08.531918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531739 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eee141-e935-4d93-ae97-6bf6222acde6" containerName="payload-processing" Oct 30 13:43:08.531918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531765 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dc3530ff-502c-4919-83b1-cc5327bc6083" containerName="payload-processing" Oct 30 13:43:08.531918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531770 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3530ff-502c-4919-83b1-cc5327bc6083" containerName="payload-processing" Oct 30 13:43:08.531918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531911 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e20eee-6850-452c-9745-3934030f7860" containerName="payload-processing" Oct 30 13:43:08.532201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531926 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47eff72-66ab-4409-9bd0-f801817d1a26" containerName="istio-proxy" Oct 30 13:43:08.532201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531937 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eee141-e935-4d93-ae97-6bf6222acde6" containerName="payload-processing" Oct 30 13:43:08.532201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.531951 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3530ff-502c-4919-83b1-cc5327bc6083" containerName="payload-processing" Oct 30 13:43:08.537502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.537480 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.553704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.553667 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp"] Oct 30 13:43:08.561454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.561418 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-939c214711aa\"" Oct 30 13:43:08.599652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.599617 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcv9\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.599820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.599793 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.599869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.599851 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.599912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.599879 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.701795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.701710 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcv9\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.701997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.701844 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.701997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.701889 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.701997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.701908 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.702730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.702699 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.702866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.702788 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.704666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.704645 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.720875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.720848 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcv9\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:08.852802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:08.852773 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:09.182024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:09.181998 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp"] Oct 30 13:43:09.182501 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:43:09.182480 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8471f82d_9ce6_4770_9729_44a7d430b482.slice/crio-2ca43267e5d5ba2a69948a11ab1bcd7a591300b0766128836af4dac5cefd2438 WatchSource:0}: Error finding container 2ca43267e5d5ba2a69948a11ab1bcd7a591300b0766128836af4dac5cefd2438: Status 404 returned error can't find the container with id 2ca43267e5d5ba2a69948a11ab1bcd7a591300b0766128836af4dac5cefd2438 Oct 30 13:43:09.594767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:09.594733 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:09.731612 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:09.731571 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:10.112364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:10.112325 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" event={"ID":"8471f82d-9ce6-4770-9729-44a7d430b482","Type":"ContainerStarted","Data":"8ebe463bdb7859cd42561383369054b0d5ba2d5de223e04893de30cd8b01cff0"} Oct 30 13:43:10.112646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:10.112511 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:10.112646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:10.112559 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" event={"ID":"8471f82d-9ce6-4770-9729-44a7d430b482","Type":"ContainerStarted","Data":"2ca43267e5d5ba2a69948a11ab1bcd7a591300b0766128836af4dac5cefd2438"} Oct 30 13:43:10.153497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:10.153450 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" podStartSLOduration=2.153435785 podStartE2EDuration="2.153435785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:10.150461099 +0000 UTC m=+2366.017079274" watchObservedRunningTime="2026-10-30 13:43:10.153435785 +0000 UTC m=+2366.020054317" Oct 30 13:43:10.752550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:10.752499 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:12.553735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:12.553700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:13.753659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:13.753624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:13.929752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:13.929719 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:14.374588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:14.374551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:14.909409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:14.909368 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd"] Oct 30 13:43:14.914505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:14.914482 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:14.947418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:14.947383 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd"] Oct 30 13:43:15.019560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.019520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5nb\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.019892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.019866 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.020011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.019993 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.020053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.020030 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.122016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.121977 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.122190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.122090 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.122190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.122122 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.122282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.122191 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5nb\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.122825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.122798 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.122960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.122940 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.124537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.124515 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.141146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.141114 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5nb\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.230461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.230432 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:15.551891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:15.551864 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd"] Oct 30 13:43:15.554339 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:43:15.554308 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e509c84_e23a_4174_b33b_95388e9d11a9.slice/crio-11433315fc2d15092ec6eec5575f5a4a3604333f30617304e3bfa26bb7554638 WatchSource:0}: Error finding container 11433315fc2d15092ec6eec5575f5a4a3604333f30617304e3bfa26bb7554638: Status 404 returned error can't find the container with id 11433315fc2d15092ec6eec5575f5a4a3604333f30617304e3bfa26bb7554638 Oct 30 13:43:16.072683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:16.072640 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:16.161009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:16.160962 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" event={"ID":"5e509c84-e23a-4174-b33b-95388e9d11a9","Type":"ContainerStarted","Data":"6c3a29865e9cfe6212ca57bacf0b8189ac3629184eb16e83f26a31a13e55022a"} Oct 30 13:43:16.161213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:16.161056 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:16.161213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:16.161074 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" event={"ID":"5e509c84-e23a-4174-b33b-95388e9d11a9","Type":"ContainerStarted","Data":"11433315fc2d15092ec6eec5575f5a4a3604333f30617304e3bfa26bb7554638"} Oct 30 13:43:16.201768 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:16.201707 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" podStartSLOduration=2.201687428 podStartE2EDuration="2.201687428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:16.200189312 +0000 UTC m=+2372.066807487" watchObservedRunningTime="2026-10-30 13:43:16.201687428 +0000 UTC m=+2372.068305601" Oct 30 13:43:16.250860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:16.250823 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:17.334694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:17.332492 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:17.537974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:17.537938 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:18.214359 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:18.214310 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:18.424996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:18.424942 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:19.952916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:19.952883 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:21.124824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:21.124783 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:43:21.211136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:21.211096 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9"] Oct 30 13:43:21.641917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:21.641864 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:22.412338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:22.412300 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:22.560322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:22.560284 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:23.057514 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:43:23.057476 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:43:23.343649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:23.343516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:24.915130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:24.915095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:26.211502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.211422 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" podUID="3ee2d183-6964-4565-ba84-3fad2f938fd7" containerName="payload-processing" containerID="cri-o://4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3" gracePeriod=25 Oct 30 13:43:26.483787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.483758 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:43:26.583962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.583925 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume\") pod \"3ee2d183-6964-4565-ba84-3fad2f938fd7\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " Oct 30 13:43:26.584163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584006 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume\") pod \"3ee2d183-6964-4565-ba84-3fad2f938fd7\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " Oct 30 13:43:26.584163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr\") pod \"3ee2d183-6964-4565-ba84-3fad2f938fd7\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " Oct 30 13:43:26.584295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584163 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials\") pod \"3ee2d183-6964-4565-ba84-3fad2f938fd7\" (UID: \"3ee2d183-6964-4565-ba84-3fad2f938fd7\") " Oct 30 13:43:26.584365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584341 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume" pod "3ee2d183-6964-4565-ba84-3fad2f938fd7" (UID: "3ee2d183-6964-4565-ba84-3fad2f938fd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:26.584472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584443 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume" pod "3ee2d183-6964-4565-ba84-3fad2f938fd7" (UID: "3ee2d183-6964-4565-ba84-3fad2f938fd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:26.584739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584720 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.584739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.584740 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ee2d183-6964-4565-ba84-3fad2f938fd7-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.586407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.586378 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr" pod "3ee2d183-6964-4565-ba84-3fad2f938fd7" (UID: "3ee2d183-6964-4565-ba84-3fad2f938fd7"). InnerVolumeSpecName "kube-api-access-k8pxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:26.586546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.586507 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials" pod "3ee2d183-6964-4565-ba84-3fad2f938fd7" (UID: "3ee2d183-6964-4565-ba84-3fad2f938fd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:26.689704 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.685556 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:26.703865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.702384 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pxr\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-kube-api-access-k8pxr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.703865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:26.702420 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ee2d183-6964-4565-ba84-3fad2f938fd7-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:27.172700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.172665 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:43:27.262838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.262809 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc"] Oct 30 13:43:27.268215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.268183 5693 generic.go:356] "Generic (PLEG): container finished" podID="3ee2d183-6964-4565-ba84-3fad2f938fd7" containerID="4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3" exitCode=0 Oct 30 13:43:27.268362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.268222 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" Oct 30 13:43:27.268362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.268235 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" event={"ID":"3ee2d183-6964-4565-ba84-3fad2f938fd7","Type":"ContainerDied","Data":"4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3"} Oct 30 13:43:27.268362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.268280 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9" event={"ID":"3ee2d183-6964-4565-ba84-3fad2f938fd7","Type":"ContainerDied","Data":"d765d6ba1eb0c288f6a03e60ab8bb71f29b200e6c6147054d82c1c874715edad"} Oct 30 13:43:27.268362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.268307 5693 scope.go:122] "RemoveContainer" containerID="4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3" Oct 30 13:43:27.290274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.290254 5693 scope.go:122] "RemoveContainer" containerID="4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3" Oct 30 13:43:27.290643 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:43:27.290581 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3\": container with ID starting with 4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3 not found: ID does not exist" containerID="4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3" Oct 30 13:43:27.290707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.290657 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3"} err="failed to get container status \"4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3\": rpc error: code = NotFound desc = could not find container \"4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3\": container with ID starting with 4cf55153ed58c5409f8bc39c09cbdef26a405b1ffe0f0d61364ffea9145740d3 not found: ID does not exist" Oct 30 13:43:27.355252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.355222 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9"] Oct 30 13:43:27.358902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:27.358879 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-8468tgrx9"] Oct 30 13:43:28.256803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:28.256773 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee2d183-6964-4565-ba84-3fad2f938fd7" path="/var/lib/kubelet/pods/3ee2d183-6964-4565-ba84-3fad2f938fd7/volumes" Oct 30 13:43:28.378857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:28.378822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:30.075144 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:30.064694 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:31.806850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:31.806811 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:32.264001 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.263947 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" podUID="be3bbc46-f8e1-4815-b44d-424e94afba99" containerName="payload-processing" containerID="cri-o://13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e" gracePeriod=25 Oct 30 13:43:32.559408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.559378 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:43:32.607760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.607728 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume\") pod \"be3bbc46-f8e1-4815-b44d-424e94afba99\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " Oct 30 13:43:32.607964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.607836 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq\") pod \"be3bbc46-f8e1-4815-b44d-424e94afba99\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " Oct 30 13:43:32.607964 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.607921 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume\") pod \"be3bbc46-f8e1-4815-b44d-424e94afba99\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " Oct 30 13:43:32.608137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.607988 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials\") pod \"be3bbc46-f8e1-4815-b44d-424e94afba99\" (UID: \"be3bbc46-f8e1-4815-b44d-424e94afba99\") " Oct 30 13:43:32.608616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.608571 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume" pod "be3bbc46-f8e1-4815-b44d-424e94afba99" (UID: "be3bbc46-f8e1-4815-b44d-424e94afba99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:32.609083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.609053 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume" pod "be3bbc46-f8e1-4815-b44d-424e94afba99" (UID: "be3bbc46-f8e1-4815-b44d-424e94afba99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:32.614635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.611576 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq" pod "be3bbc46-f8e1-4815-b44d-424e94afba99" (UID: "be3bbc46-f8e1-4815-b44d-424e94afba99"). InnerVolumeSpecName "kube-api-access-8gmxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:32.614635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.611646 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials" pod "be3bbc46-f8e1-4815-b44d-424e94afba99" (UID: "be3bbc46-f8e1-4815-b44d-424e94afba99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:32.710053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.710014 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:32.710053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.710045 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmxq\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-kube-api-access-8gmxq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:32.710053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.710055 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3bbc46-f8e1-4815-b44d-424e94afba99-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:32.710310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:32.710066 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3bbc46-f8e1-4815-b44d-424e94afba99-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.316389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.316356 5693 generic.go:356] "Generic (PLEG): container finished" podID="be3bbc46-f8e1-4815-b44d-424e94afba99" containerID="13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e" exitCode=0 Oct 30 13:43:33.316906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.316399 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" event={"ID":"be3bbc46-f8e1-4815-b44d-424e94afba99","Type":"ContainerDied","Data":"13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e"} Oct 30 13:43:33.316906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.316415 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" Oct 30 13:43:33.316906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.316468 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc" event={"ID":"be3bbc46-f8e1-4815-b44d-424e94afba99","Type":"ContainerDied","Data":"4ea274d49653e072600a0af870c348e284e10e059d6ca4cc546c3f039a9b8a49"} Oct 30 13:43:33.316906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.316496 5693 scope.go:122] "RemoveContainer" containerID="13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e" Oct 30 13:43:33.328299 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.328278 5693 scope.go:122] "RemoveContainer" containerID="13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e" Oct 30 13:43:33.328577 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:43:33.328557 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e\": container with ID starting with 13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e not found: ID does not exist" containerID="13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e" Oct 30 13:43:33.329550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.328589 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e"} err="failed to get container status \"13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e\": rpc error: code = NotFound desc = could not find container \"13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e\": container with ID starting with 13d82dd536d3f9756b3413ffe1b4b710f221e35d5a210405c4cce1275634221e not found: ID does not exist" Oct 30 13:43:33.368745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.368719 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc"] Oct 30 13:43:33.380395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.380372 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-79946f7zc"] Oct 30 13:43:33.526624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:33.525827 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:34.268104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:34.268069 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3bbc46-f8e1-4815-b44d-424e94afba99" path="/var/lib/kubelet/pods/be3bbc46-f8e1-4815-b44d-424e94afba99/volumes" Oct 30 13:43:35.221621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:35.214889 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:37.048635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:37.044941 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:38.770636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:38.770582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:40.574547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:40.574511 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:42.329124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:42.329073 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:43.942888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:43.942859 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:45.682015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:45.681973 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:47.453552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:47.453516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:49.180326 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:49.180231 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:50.976318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:50.976286 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:52.741742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:52.741673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:53.056032 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:43:53.055949 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:43:54.391335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:54.391247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.073812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:56.073771 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.258680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:56.258645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.456738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:56.456650 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.622073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:56.622006 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.772714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:56.772681 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:57.536417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:57.536376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:57.778514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:57.778481 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:57.994178 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:57.994145 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:58.177584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:58.177524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:58.661573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:58.661539 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:58.851911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:43:58.851877 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:00.037765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.037732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:00.235403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.235371 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx"] Oct 30 13:44:00.236548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.236520 5693 cpu_manager.go:414] "containerMap: removing container" podUID="be3bbc46-f8e1-4815-b44d-424e94afba99" containerName="payload-processing" Oct 30 13:44:00.236548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.236547 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3bbc46-f8e1-4815-b44d-424e94afba99" containerName="payload-processing" Oct 30 13:44:00.237249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.236572 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3ee2d183-6964-4565-ba84-3fad2f938fd7" containerName="payload-processing" Oct 30 13:44:00.237249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.236579 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee2d183-6964-4565-ba84-3fad2f938fd7" containerName="payload-processing" Oct 30 13:44:00.237249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.236792 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee2d183-6964-4565-ba84-3fad2f938fd7" containerName="payload-processing" Oct 30 13:44:00.237249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.236831 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3bbc46-f8e1-4815-b44d-424e94afba99" containerName="payload-processing" Oct 30 13:44:00.242024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.241996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:00.242866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.242847 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.263361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.263333 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.263481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.263416 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmr7\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.263560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.263538 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.263667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.263650 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.266806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.266784 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-6feb6713504c\"" Oct 30 13:44:00.279381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.279352 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-7fjrf\"" Oct 30 13:44:00.279790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.279452 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"openshift-service-ca.crt\"" Oct 30 13:44:00.279790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.279533 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"routing-overlay\"" Oct 30 13:44:00.279790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.279571 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cad2c6\"" Oct 30 13:44:00.280198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.280178 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"kube-root-ca.crt\"" Oct 30 13:44:00.284874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.282919 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx"] Oct 30 13:44:00.365442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.365348 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.365628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.365481 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.365687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.365638 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.365755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.365735 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmr7\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.366225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.366199 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.366342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.366266 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.368189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.368155 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.387279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.387249 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmr7\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.408645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.408583 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:00.585186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.581297 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:00.645863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.645063 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8"] Oct 30 13:44:00.653910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.653058 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.677752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.677166 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cw6t5\"" Oct 30 13:44:00.680880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.680846 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh"] Oct 30 13:44:00.692796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.691926 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8"] Oct 30 13:44:00.692796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.692047 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696076 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5295dc-10f1-476e-b269-54c956fb9490-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696141 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696203 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696312 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696367 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f485m\" (UniqueName: \"kubernetes.io/projected/2a5295dc-10f1-476e-b269-54c956fb9490-kube-api-access-f485m\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696447 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5295dc-10f1-476e-b269-54c956fb9490-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696572 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.703283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.696622 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5295dc-10f1-476e-b269-54c956fb9490-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.746675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.744166 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh"] Oct 30 13:44:00.797983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.797927 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798051 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798085 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798113 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798146 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5295dc-10f1-476e-b269-54c956fb9490-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798206 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798261 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798286 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5295dc-10f1-476e-b269-54c956fb9490-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798363 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798385 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5295dc-10f1-476e-b269-54c956fb9490-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798419 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798439 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798488 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.798858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798519 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798569 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.798858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798617 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.799141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.799075 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.799535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.799510 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.799675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.799661 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.799771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.799748 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5295dc-10f1-476e-b269-54c956fb9490-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.799834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.799808 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.799925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.798671 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f485m\" (UniqueName: \"kubernetes.io/projected/2a5295dc-10f1-476e-b269-54c956fb9490-kube-api-access-f485m\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.800074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.800054 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdqth\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.802061 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.802037 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5295dc-10f1-476e-b269-54c956fb9490-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.802242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.802224 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5295dc-10f1-476e-b269-54c956fb9490-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.823425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.823401 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5295dc-10f1-476e-b269-54c956fb9490-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.823673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.823655 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f485m\" (UniqueName: \"kubernetes.io/projected/2a5295dc-10f1-476e-b269-54c956fb9490-kube-api-access-f485m\") pod \"maas-default-gateway-openshift-default-67bf986f99-dhjq8\" (UID: \"2a5295dc-10f1-476e-b269-54c956fb9490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:00.901488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901406 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.901488 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901464 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.901744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901543 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.901744 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901648 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.901926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901744 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdqth\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.901926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901783 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.901926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901907 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.902215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901826 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.902215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901966 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.902215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.901996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.902427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.902349 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.903115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.903084 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.903221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.903131 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.903279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.903234 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.905300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.905272 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.905930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.905902 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.922717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.922688 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdqth\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.923419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.923393 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-52qkh\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:00.958338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.958313 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx"] Oct 30 13:44:00.960431 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:00.960404 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50295159_ec8a_4bae_bb53_a83a575aca2d.slice/crio-5b2d87aab2662d52b16fc148b10a981694fa327f217affb2b53288819c95a59f WatchSource:0}: Error finding container 5b2d87aab2662d52b16fc148b10a981694fa327f217affb2b53288819c95a59f: Status 404 returned error can't find the container with id 5b2d87aab2662d52b16fc148b10a981694fa327f217affb2b53288819c95a59f Oct 30 13:44:00.978544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:00.978519 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:01.075659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.074907 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:01.338666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.338639 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8"] Oct 30 13:44:01.339896 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:01.339869 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5295dc_10f1_476e_b269_54c956fb9490.slice/crio-1eebb3ef1cc47bf816d00a6fd3ba26da6e611494e9608e16e6ffa3271dd0ea38 WatchSource:0}: Error finding container 1eebb3ef1cc47bf816d00a6fd3ba26da6e611494e9608e16e6ffa3271dd0ea38: Status 404 returned error can't find the container with id 1eebb3ef1cc47bf816d00a6fd3ba26da6e611494e9608e16e6ffa3271dd0ea38 Oct 30 13:44:01.343012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.342973 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:01.343113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.343042 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:01.343113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.343079 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:01.422484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.422443 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh"] Oct 30 13:44:01.424474 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:01.424446 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e228f8_06d6_425b_baf5_ee05cc01130c.slice/crio-94cf7a6fd0e1b3d561037671bdbe87e95c9c1cc3dd30cbf2140e00d74245083d WatchSource:0}: Error finding container 94cf7a6fd0e1b3d561037671bdbe87e95c9c1cc3dd30cbf2140e00d74245083d: Status 404 returned error can't find the container with id 94cf7a6fd0e1b3d561037671bdbe87e95c9c1cc3dd30cbf2140e00d74245083d Oct 30 13:44:01.427999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.427963 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:01.428125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.428047 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:01.428125 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.428094 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:01.620797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.620655 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" event={"ID":"91e228f8-06d6-425b-baf5-ee05cc01130c","Type":"ContainerStarted","Data":"2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05"} Oct 30 13:44:01.620797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.620727 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" event={"ID":"91e228f8-06d6-425b-baf5-ee05cc01130c","Type":"ContainerStarted","Data":"94cf7a6fd0e1b3d561037671bdbe87e95c9c1cc3dd30cbf2140e00d74245083d"} Oct 30 13:44:01.623700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.623663 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" event={"ID":"2a5295dc-10f1-476e-b269-54c956fb9490","Type":"ContainerStarted","Data":"bd445ccf55f1e421011a2b6c2e9ac718cb32e7a40983b6218480ad09280fc8ca"} Oct 30 13:44:01.623861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.623726 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" event={"ID":"2a5295dc-10f1-476e-b269-54c956fb9490","Type":"ContainerStarted","Data":"1eebb3ef1cc47bf816d00a6fd3ba26da6e611494e9608e16e6ffa3271dd0ea38"} Oct 30 13:44:01.626886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.626310 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" event={"ID":"50295159-ec8a-4bae-bb53-a83a575aca2d","Type":"ContainerStarted","Data":"d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0"} Oct 30 13:44:01.626886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.626364 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" event={"ID":"50295159-ec8a-4bae-bb53-a83a575aca2d","Type":"ContainerStarted","Data":"5b2d87aab2662d52b16fc148b10a981694fa327f217affb2b53288819c95a59f"} Oct 30 13:44:01.626886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.626408 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:01.666842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.666748 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podStartSLOduration=1.66672679 podStartE2EDuration="1.66672679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:01.665206665 +0000 UTC m=+2417.531824831" watchObservedRunningTime="2026-10-30 13:44:01.66672679 +0000 UTC m=+2417.533344964" Oct 30 13:44:01.722387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.722322 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" podStartSLOduration=1.722300798 podStartE2EDuration="1.722300798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:01.717118995 +0000 UTC m=+2417.583737172" watchObservedRunningTime="2026-10-30 13:44:01.722300798 +0000 UTC m=+2417.588918972" Oct 30 13:44:01.762498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.762437 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podStartSLOduration=1.76242272 podStartE2EDuration="1.76242272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:01.75809181 +0000 UTC m=+2417.624709986" watchObservedRunningTime="2026-10-30 13:44:01.76242272 +0000 UTC m=+2417.629040893" Oct 30 13:44:01.979637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:01.979585 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:02.075653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:02.075612 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:02.077095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:02.077071 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:02.077417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:02.077122 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:02.325718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:02.322432 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:02.980823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:02.980775 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:02.981015 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:02.980847 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:03.076149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:03.076110 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:03.076373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:03.076176 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:03.979657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:03.979010 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:03.979657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:03.979074 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:04.077619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:04.077067 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:04.077619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:04.077123 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:04.211348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:04.208161 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:04.979559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:04.979011 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:04.979559 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:04.979077 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:05.077103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:05.077060 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:05.077404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:05.077371 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:05.979980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:05.979930 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:05.980426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:05.979998 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:06.077621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:06.077232 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:06.077621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:06.077295 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:06.178289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:06.178260 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:06.979840 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:06.979806 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:06.980059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:06.980038 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:07.077946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:07.077907 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:07.078262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:07.078236 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:07.979096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:07.979055 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:07.979315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:07.979124 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:08.075923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.075882 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:08.076447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.075935 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:08.251676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.245587 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:08.558222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.558128 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd"] Oct 30 13:44:08.614253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.614216 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:08.760306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.760275 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:08.985621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.981963 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dhjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:44:08.985621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:08.982022 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" podUID="2a5295dc-10f1-476e-b269-54c956fb9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 13:44:09.080628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:09.076948 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-52qkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:44:09.080628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:09.077007 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:44:09.270576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:09.270470 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.369159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.369120 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.494312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.494261 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb"] Oct 30 13:44:10.625535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.625366 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.735642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.735592 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:10.746951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.746923 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:10.774224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.774189 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dhjq8" Oct 30 13:44:10.908749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.901188 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:10.908749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.904907 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.910614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.910571 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:10.918052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:10.915939 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:44:11.093656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:11.093572 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:11.461079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:11.461043 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:11.640333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:11.640294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:11.895081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:11.895045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.248735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:12.248705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.445451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:12.445406 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.685972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:12.685898 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:12.791038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:12.790967 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.980991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:12.980960 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:13.170462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.170430 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:13.459999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.459909 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx"] Oct 30 13:44:13.558693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.558654 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" podUID="5e509c84-e23a-4174-b33b-95388e9d11a9" containerName="payload-processing" containerID="cri-o://6c3a29865e9cfe6212ca57bacf0b8189ac3629184eb16e83f26a31a13e55022a" gracePeriod=25 Oct 30 13:44:13.866082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.866049 5693 generic.go:356] "Generic (PLEG): container finished" podID="5e509c84-e23a-4174-b33b-95388e9d11a9" containerID="6c3a29865e9cfe6212ca57bacf0b8189ac3629184eb16e83f26a31a13e55022a" exitCode=0 Oct 30 13:44:13.866685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.866131 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" event={"ID":"5e509c84-e23a-4174-b33b-95388e9d11a9","Type":"ContainerDied","Data":"6c3a29865e9cfe6212ca57bacf0b8189ac3629184eb16e83f26a31a13e55022a"} Oct 30 13:44:13.866685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.866199 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" event={"ID":"5e509c84-e23a-4174-b33b-95388e9d11a9","Type":"ContainerDied","Data":"11433315fc2d15092ec6eec5575f5a4a3604333f30617304e3bfa26bb7554638"} Oct 30 13:44:13.866685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.866230 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11433315fc2d15092ec6eec5575f5a4a3604333f30617304e3bfa26bb7554638" Oct 30 13:44:13.875553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.875522 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:44:13.932307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.932267 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume\") pod \"5e509c84-e23a-4174-b33b-95388e9d11a9\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " Oct 30 13:44:13.932500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.932360 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials\") pod \"5e509c84-e23a-4174-b33b-95388e9d11a9\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " Oct 30 13:44:13.932500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.932437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb\") pod \"5e509c84-e23a-4174-b33b-95388e9d11a9\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " Oct 30 13:44:13.932731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.932563 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume\") pod \"5e509c84-e23a-4174-b33b-95388e9d11a9\" (UID: \"5e509c84-e23a-4174-b33b-95388e9d11a9\") " Oct 30 13:44:13.933742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.933710 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume" pod "5e509c84-e23a-4174-b33b-95388e9d11a9" (UID: "5e509c84-e23a-4174-b33b-95388e9d11a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:13.934474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.934206 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume" pod "5e509c84-e23a-4174-b33b-95388e9d11a9" (UID: "5e509c84-e23a-4174-b33b-95388e9d11a9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:13.937947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.937924 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials" pod "5e509c84-e23a-4174-b33b-95388e9d11a9" (UID: "5e509c84-e23a-4174-b33b-95388e9d11a9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:13.939157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:13.939131 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb" pod "5e509c84-e23a-4174-b33b-95388e9d11a9" (UID: "5e509c84-e23a-4174-b33b-95388e9d11a9"). InnerVolumeSpecName "kube-api-access-mm5nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:14.034857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.034761 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.034857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.034799 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.034857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.034815 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm5nb\" (UniqueName: \"kubernetes.io/projected/5e509c84-e23a-4174-b33b-95388e9d11a9-kube-api-access-mm5nb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.034857 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.034832 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e509c84-e23a-4174-b33b-95388e9d11a9-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.151362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.151330 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:14.704383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.704236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:14.886790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.886702 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd" Oct 30 13:44:14.986289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:14.986247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:15.057677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.057642 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd"] Oct 30 13:44:15.069222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.069174 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fg6hjd"] Oct 30 13:44:15.495941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.495881 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" podUID="77d1f9d5-b522-4ed6-a28f-0103369c174b" containerName="payload-processing" containerID="cri-o://0bb27b9be0486efef217fbfddf91fc731bef2614dcfdfae5f20b339417cb548f" gracePeriod=25 Oct 30 13:44:15.626738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.622376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:15.705961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.705919 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n"] Oct 30 13:44:15.707879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.707825 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5e509c84-e23a-4174-b33b-95388e9d11a9" containerName="payload-processing" Oct 30 13:44:15.708127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.708099 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e509c84-e23a-4174-b33b-95388e9d11a9" containerName="payload-processing" Oct 30 13:44:15.708406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.708386 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e509c84-e23a-4174-b33b-95388e9d11a9" containerName="payload-processing" Oct 30 13:44:15.712902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.712878 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811463 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811539 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsftv\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811706 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811738 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811765 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811797 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.811880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811831 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.812063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811883 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.812063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.811957 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.883012 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.880075 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n"] Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.914921 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915087 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915139 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsftv\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915289 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915316 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915346 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915381 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915416 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915415 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.915777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.916054 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.916440 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.917113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.916760 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.918451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.917498 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.919960 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.919873 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:15.924321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.922547 5693 generic.go:356] "Generic (PLEG): container finished" podID="77d1f9d5-b522-4ed6-a28f-0103369c174b" containerID="0bb27b9be0486efef217fbfddf91fc731bef2614dcfdfae5f20b339417cb548f" exitCode=0 Oct 30 13:44:15.924321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.922691 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" event={"ID":"77d1f9d5-b522-4ed6-a28f-0103369c174b","Type":"ContainerDied","Data":"0bb27b9be0486efef217fbfddf91fc731bef2614dcfdfae5f20b339417cb548f"} Oct 30 13:44:15.924321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:15.924171 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:16.006985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.006909 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:16.007801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.007731 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsftv\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv\") pod \"maas-default-gateway-openshift-default-67bf986f99-8g88n\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:16.071714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.063325 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:44:16.119014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.118973 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4\") pod \"77d1f9d5-b522-4ed6-a28f-0103369c174b\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " Oct 30 13:44:16.119219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.119027 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume\") pod \"77d1f9d5-b522-4ed6-a28f-0103369c174b\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " Oct 30 13:44:16.119219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.119155 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials\") pod \"77d1f9d5-b522-4ed6-a28f-0103369c174b\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " Oct 30 13:44:16.120260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.119405 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume\") pod \"77d1f9d5-b522-4ed6-a28f-0103369c174b\" (UID: \"77d1f9d5-b522-4ed6-a28f-0103369c174b\") " Oct 30 13:44:16.120260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.119664 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume" pod "77d1f9d5-b522-4ed6-a28f-0103369c174b" (UID: "77d1f9d5-b522-4ed6-a28f-0103369c174b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:16.120260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.120044 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume" pod "77d1f9d5-b522-4ed6-a28f-0103369c174b" (UID: "77d1f9d5-b522-4ed6-a28f-0103369c174b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:16.120260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.120214 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:16.120260 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.120233 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77d1f9d5-b522-4ed6-a28f-0103369c174b-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:16.123563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.123523 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials" pod "77d1f9d5-b522-4ed6-a28f-0103369c174b" (UID: "77d1f9d5-b522-4ed6-a28f-0103369c174b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:16.123762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.123678 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4" pod "77d1f9d5-b522-4ed6-a28f-0103369c174b" (UID: "77d1f9d5-b522-4ed6-a28f-0103369c174b"). InnerVolumeSpecName "kube-api-access-9dtn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:16.185924 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.180747 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:16.225124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.223087 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-kube-api-access-9dtn4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:16.225124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.223125 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77d1f9d5-b522-4ed6-a28f-0103369c174b-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:16.302216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.302171 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e509c84-e23a-4174-b33b-95388e9d11a9" path="/var/lib/kubelet/pods/5e509c84-e23a-4174-b33b-95388e9d11a9/volumes" Oct 30 13:44:16.633282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.631641 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n"] Oct 30 13:44:16.637232 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:16.637155 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e18e077_bc58_4739_b0e8_1c6117524ece.slice/crio-c275d5786c9fa7a044512283d5b35ce7b865f54ea8bda25f23e21a9c732d86c1 WatchSource:0}: Error finding container c275d5786c9fa7a044512283d5b35ce7b865f54ea8bda25f23e21a9c732d86c1: Status 404 returned error can't find the container with id c275d5786c9fa7a044512283d5b35ce7b865f54ea8bda25f23e21a9c732d86c1 Oct 30 13:44:16.641110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.641071 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:16.641212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.641160 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:16.641287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.641210 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:16.929440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.929368 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:16.932879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.932847 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" event={"ID":"1e18e077-bc58-4739-b0e8-1c6117524ece","Type":"ContainerStarted","Data":"59d55433673608f2effd21df4dd3b9c906c330b948210cbb884e07621fe571aa"} Oct 30 13:44:16.933030 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.932908 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" event={"ID":"1e18e077-bc58-4739-b0e8-1c6117524ece","Type":"ContainerStarted","Data":"c275d5786c9fa7a044512283d5b35ce7b865f54ea8bda25f23e21a9c732d86c1"} Oct 30 13:44:16.934302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.934281 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" Oct 30 13:44:16.934422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.934318 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb" event={"ID":"77d1f9d5-b522-4ed6-a28f-0103369c174b","Type":"ContainerDied","Data":"6b2847d82db33607c0fac8a775dfb8e0bd8c8e8d6bb539141aaf20bda9f20019"} Oct 30 13:44:16.934422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:16.934370 5693 scope.go:122] "RemoveContainer" containerID="0bb27b9be0486efef217fbfddf91fc731bef2614dcfdfae5f20b339417cb548f" Oct 30 13:44:17.065622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.065185 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podStartSLOduration=2.065162545 podStartE2EDuration="2.065162545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:16.990974184 +0000 UTC m=+2432.857592359" watchObservedRunningTime="2026-10-30 13:44:17.065162545 +0000 UTC m=+2432.931780719" Oct 30 13:44:17.071579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.071513 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb"] Oct 30 13:44:17.086508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.085665 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-5b8bvjgdb"] Oct 30 13:44:17.161522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.161487 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj"] Oct 30 13:44:17.162287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.162272 5693 cpu_manager.go:414] "containerMap: removing container" podUID="77d1f9d5-b522-4ed6-a28f-0103369c174b" containerName="payload-processing" Oct 30 13:44:17.162338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.162289 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d1f9d5-b522-4ed6-a28f-0103369c174b" containerName="payload-processing" Oct 30 13:44:17.162766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.162413 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d1f9d5-b522-4ed6-a28f-0103369c174b" containerName="payload-processing" Oct 30 13:44:17.167496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.167476 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.182055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.181938 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:17.183743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.183455 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:17.183743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.183521 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:17.185535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.185493 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-046f0a08f638\"" Oct 30 13:44:17.185700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.185675 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wmmjw\"" Oct 30 13:44:17.186381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.186363 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 13:44:17.186507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.186462 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 13:44:17.186579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.186526 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 13:44:17.186681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.186625 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 13:44:17.187526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.187505 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj"] Oct 30 13:44:17.246160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.246126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.246344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.246194 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.246436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.246377 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlxm\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.246506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.246484 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.378618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.375779 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.378618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.375978 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.378618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.376086 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.378618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.376255 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlxm\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.378618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.377778 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.379617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.379263 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.395695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.388876 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.450668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.445459 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlxm\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm\") pod \"payload-processing-external-model-b7d6584f7-nn7qj\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.492835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.492800 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:17.819657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.816448 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:17.873585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.873554 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj"] Oct 30 13:44:17.875345 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:17.875313 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf64d38_b588_43c7_8344_f6b804961b2d.slice/crio-4cf6f55ded996b6a586918b68c68018d63b58ff668e3f4517dc06728e90f6b3a WatchSource:0}: Error finding container 4cf6f55ded996b6a586918b68c68018d63b58ff668e3f4517dc06728e90f6b3a: Status 404 returned error can't find the container with id 4cf6f55ded996b6a586918b68c68018d63b58ff668e3f4517dc06728e90f6b3a Oct 30 13:44:17.959489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:17.959456 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" event={"ID":"2bf64d38-b588-43c7-8344-f6b804961b2d","Type":"ContainerStarted","Data":"4cf6f55ded996b6a586918b68c68018d63b58ff668e3f4517dc06728e90f6b3a"} Oct 30 13:44:18.000415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.000391 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:18.182182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.182099 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:18.182182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.182165 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:18.277876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.277840 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d1f9d5-b522-4ed6-a28f-0103369c174b" path="/var/lib/kubelet/pods/77d1f9d5-b522-4ed6-a28f-0103369c174b/volumes" Oct 30 13:44:18.460582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.460486 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" podUID="50295159-ec8a-4bae-bb53-a83a575aca2d" containerName="payload-processing" containerID="cri-o://d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0" gracePeriod=25 Oct 30 13:44:18.801739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.801710 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:18.915957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.915915 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials\") pod \"50295159-ec8a-4bae-bb53-a83a575aca2d\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " Oct 30 13:44:18.916162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.916050 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume\") pod \"50295159-ec8a-4bae-bb53-a83a575aca2d\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " Oct 30 13:44:18.916162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.916116 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume\") pod \"50295159-ec8a-4bae-bb53-a83a575aca2d\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " Oct 30 13:44:18.916282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.916184 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7\") pod \"50295159-ec8a-4bae-bb53-a83a575aca2d\" (UID: \"50295159-ec8a-4bae-bb53-a83a575aca2d\") " Oct 30 13:44:18.916517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.916479 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume" pod "50295159-ec8a-4bae-bb53-a83a575aca2d" (UID: "50295159-ec8a-4bae-bb53-a83a575aca2d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:18.916671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.916584 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume" pod "50295159-ec8a-4bae-bb53-a83a575aca2d" (UID: "50295159-ec8a-4bae-bb53-a83a575aca2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:18.917003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.916982 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:18.917069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.917010 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50295159-ec8a-4bae-bb53-a83a575aca2d-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:18.918373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.918348 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials" pod "50295159-ec8a-4bae-bb53-a83a575aca2d" (UID: "50295159-ec8a-4bae-bb53-a83a575aca2d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:18.918487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.918436 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7" pod "50295159-ec8a-4bae-bb53-a83a575aca2d" (UID: "50295159-ec8a-4bae-bb53-a83a575aca2d"). InnerVolumeSpecName "kube-api-access-zqmr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:18.970809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.970770 5693 generic.go:356] "Generic (PLEG): container finished" podID="50295159-ec8a-4bae-bb53-a83a575aca2d" containerID="d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0" exitCode=0 Oct 30 13:44:18.971247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.970824 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" Oct 30 13:44:18.971247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.970856 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" event={"ID":"50295159-ec8a-4bae-bb53-a83a575aca2d","Type":"ContainerDied","Data":"d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0"} Oct 30 13:44:18.971247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.970946 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx" event={"ID":"50295159-ec8a-4bae-bb53-a83a575aca2d","Type":"ContainerDied","Data":"5b2d87aab2662d52b16fc148b10a981694fa327f217affb2b53288819c95a59f"} Oct 30 13:44:18.971247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.970978 5693 scope.go:122] "RemoveContainer" containerID="d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0" Oct 30 13:44:18.987976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.987953 5693 scope.go:122] "RemoveContainer" containerID="d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0" Oct 30 13:44:18.988096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.988076 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" event={"ID":"2bf64d38-b588-43c7-8344-f6b804961b2d","Type":"ContainerStarted","Data":"a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6"} Oct 30 13:44:18.988574 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:18.988543 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0\": container with ID starting with d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0 not found: ID does not exist" containerID="d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0" Oct 30 13:44:18.988574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:18.988581 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0"} err="failed to get container status \"d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0\": rpc error: code = NotFound desc = could not find container \"d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0\": container with ID starting with d9ae4d0589f5e09ab7e5c69b4834a9d062ca521d553a45bc329e0251a833e6c0 not found: ID does not exist" Oct 30 13:44:19.019024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.018945 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmr7\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-kube-api-access-zqmr7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:19.019024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.018973 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50295159-ec8a-4bae-bb53-a83a575aca2d-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:19.093860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.093809 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" podStartSLOduration=2.093795142 podStartE2EDuration="2.093795142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:19.049131638 +0000 UTC m=+2434.915749816" watchObservedRunningTime="2026-10-30 13:44:19.093795142 +0000 UTC m=+2434.960413328" Oct 30 13:44:19.094078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.094061 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx"] Oct 30 13:44:19.108327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.108298 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcf56dx"] Oct 30 13:44:19.182050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.182005 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:19.182238 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.182085 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:19.667624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.666929 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:19.996560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:19.996533 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:20.182082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:20.182043 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:20.182269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:20.182093 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:20.265217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:20.265139 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50295159-ec8a-4bae-bb53-a83a575aca2d" path="/var/lib/kubelet/pods/50295159-ec8a-4bae-bb53-a83a575aca2d/volumes" Oct 30 13:44:21.179057 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:21.179023 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-046f0a08f638: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:21.179057 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:21.179051 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:21.179655 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:21.179137 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials podName:2bf64d38-b588-43c7-8344-f6b804961b2d nodeName:}" failed. No retries permitted until 2026-10-30 13:44:21.679113511 +0000 UTC m=+2437.545731668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials") pod "payload-processing-external-model-b7d6584f7-nn7qj" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d") : secret "agc-cred-046f0a08f638" not found Oct 30 13:44:21.181438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:21.181396 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:21.181568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:21.181450 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:21.191182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:21.191150 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj"] Oct 30 13:44:21.194542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:21.194516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:21.453010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:21.452919 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:21.688639 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:21.688590 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-046f0a08f638: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:21.688880 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:21.688866 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:21.689052 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:21.689039 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials podName:2bf64d38-b588-43c7-8344-f6b804961b2d nodeName:}" failed. No retries permitted until 2026-10-30 13:44:22.68901526 +0000 UTC m=+2438.555633469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials") pod "payload-processing-external-model-b7d6584f7-nn7qj" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d") : secret "agc-cred-046f0a08f638" not found Oct 30 13:44:22.014784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:22.014756 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:22.134662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:22.132207 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:22.191251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:22.188142 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:22.191251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:22.188206 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:22.706415 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:22.706372 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-046f0a08f638: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:22.706415 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:22.706409 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:22.706672 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:22.706470 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials podName:2bf64d38-b588-43c7-8344-f6b804961b2d nodeName:}" failed. No retries permitted until 2026-10-30 13:44:24.706456258 +0000 UTC m=+2440.573074409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials") pod "payload-processing-external-model-b7d6584f7-nn7qj" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d") : secret "agc-cred-046f0a08f638" not found Oct 30 13:44:23.078556 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:23.078516 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:44:23.181557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:23.181523 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:23.181762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:23.181575 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:23.969282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:23.969243 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:24.181402 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:24.181359 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8g88n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 13:44:24.181578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:24.181430 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 13:44:24.740486 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:24.740442 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-046f0a08f638: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:24.740486 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:24.740481 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj: secret "agc-cred-046f0a08f638" not found Oct 30 13:44:24.740876 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:24.740538 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials podName:2bf64d38-b588-43c7-8344-f6b804961b2d nodeName:}" failed. No retries permitted until 2026-10-30 13:44:28.74051878 +0000 UTC m=+2444.607136930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials") pod "payload-processing-external-model-b7d6584f7-nn7qj" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d") : secret "agc-cred-046f0a08f638" not found Oct 30 13:44:25.565476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:25.565440 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:25.773464 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:25.773430 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:25.773969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:25.773944 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:25.774743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:25.774722 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:44:25.785165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:25.785131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:26.054191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:26.054163 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:26.262880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:26.262844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:26.609936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:26.609892 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.011955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.011918 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" podUID="2bf64d38-b588-43c7-8344-f6b804961b2d" containerName="payload-processing" containerID="cri-o://a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6" gracePeriod=25 Oct 30 13:44:27.297978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.297904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.357526 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.357497 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:27.538265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.534234 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume\") pod \"2bf64d38-b588-43c7-8344-f6b804961b2d\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " Oct 30 13:44:27.594022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.593891 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume" pod "2bf64d38-b588-43c7-8344-f6b804961b2d" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:27.606625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.594281 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm\") pod \"2bf64d38-b588-43c7-8344-f6b804961b2d\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " Oct 30 13:44:27.607000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.606966 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume\") pod \"2bf64d38-b588-43c7-8344-f6b804961b2d\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " Oct 30 13:44:27.607189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.607171 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials\") pod \"2bf64d38-b588-43c7-8344-f6b804961b2d\" (UID: \"2bf64d38-b588-43c7-8344-f6b804961b2d\") " Oct 30 13:44:27.618697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.608142 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.759196 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.623728 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume" pod "2bf64d38-b588-43c7-8344-f6b804961b2d" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:27.790628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.788865 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm" pod "2bf64d38-b588-43c7-8344-f6b804961b2d" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d"). InnerVolumeSpecName "kube-api-access-8hlxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:27.795991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.795753 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.806785 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.806728 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials" pod "2bf64d38-b588-43c7-8344-f6b804961b2d" (UID: "2bf64d38-b588-43c7-8344-f6b804961b2d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:27.861004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.860862 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlxm\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-kube-api-access-8hlxm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.861004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.860903 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf64d38-b588-43c7-8344-f6b804961b2d-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.861004 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:27.860920 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf64d38-b588-43c7-8344-f6b804961b2d-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:28.075851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.075809 5693 generic.go:356] "Generic (PLEG): container finished" podID="2bf64d38-b588-43c7-8344-f6b804961b2d" containerID="a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6" exitCode=0 Oct 30 13:44:28.076066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.076031 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" event={"ID":"2bf64d38-b588-43c7-8344-f6b804961b2d","Type":"ContainerDied","Data":"a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6"} Oct 30 13:44:28.076138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.076100 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" event={"ID":"2bf64d38-b588-43c7-8344-f6b804961b2d","Type":"ContainerDied","Data":"4cf6f55ded996b6a586918b68c68018d63b58ff668e3f4517dc06728e90f6b3a"} Oct 30 13:44:28.076138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.076120 5693 scope.go:122] "RemoveContainer" containerID="a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6" Oct 30 13:44:28.076270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.076231 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj" Oct 30 13:44:28.097116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.097086 5693 scope.go:122] "RemoveContainer" containerID="a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6" Oct 30 13:44:28.097475 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:28.097452 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6\": container with ID starting with a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6 not found: ID does not exist" containerID="a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6" Oct 30 13:44:28.097561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.097491 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6"} err="failed to get container status \"a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6\": rpc error: code = NotFound desc = could not find container \"a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6\": container with ID starting with a7f2cf58836232ea0af0b000457a671bff5358b124e2349fe41eab570f689ac6 not found: ID does not exist" Oct 30 13:44:28.162483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.162395 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj"] Oct 30 13:44:28.169100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.169062 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-nn7qj"] Oct 30 13:44:28.241547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.241508 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:28.261611 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.261559 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf64d38-b588-43c7-8344-f6b804961b2d" path="/var/lib/kubelet/pods/2bf64d38-b588-43c7-8344-f6b804961b2d/volumes" Oct 30 13:44:28.427407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.427323 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:28.617123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.617079 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:28.898999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:28.898963 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:29.094618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.086874 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-j6662"] Oct 30 13:44:29.102621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.098917 5693 cpu_manager.go:414] "containerMap: removing container" podUID="50295159-ec8a-4bae-bb53-a83a575aca2d" containerName="payload-processing" Oct 30 13:44:29.102621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.098956 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50295159-ec8a-4bae-bb53-a83a575aca2d" containerName="payload-processing" Oct 30 13:44:29.102621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.098985 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2bf64d38-b588-43c7-8344-f6b804961b2d" containerName="payload-processing" Oct 30 13:44:29.102621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.098992 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf64d38-b588-43c7-8344-f6b804961b2d" containerName="payload-processing" Oct 30 13:44:29.102621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.099237 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="50295159-ec8a-4bae-bb53-a83a575aca2d" containerName="payload-processing" Oct 30 13:44:29.102621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.099251 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf64d38-b588-43c7-8344-f6b804961b2d" containerName="payload-processing" Oct 30 13:44:29.137644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.134999 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.137644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.137143 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-j6662"] Oct 30 13:44:29.153240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.153160 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gwdw4\"" Oct 30 13:44:29.168266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.168085 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 13:44:29.168266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.168126 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-046f0a08f638\"" Oct 30 13:44:29.168266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.168149 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 13:44:29.168266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.168194 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 13:44:29.168266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.168233 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 13:44:29.230285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.230235 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8r4\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.230693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.230666 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.230890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.230874 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.231159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.231119 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.369995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.369912 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8r4\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.387156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.387122 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.388834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.388799 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.388991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.388970 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.390807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.390425 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.391374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.391357 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.401638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.398351 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.405339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.405274 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8r4\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4\") pod \"payload-processing-external-model-b7d6584f7-j6662\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.478758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.478723 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:29.794876 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.794843 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-j6662"] Oct 30 13:44:29.797316 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:29.797283 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0d9328_129f_4911_97a0_27ed8e5b11f4.slice/crio-f6622ba5761556115ddcab0651c453003ff147fac125fc2c0471131fd0437ebc WatchSource:0}: Error finding container f6622ba5761556115ddcab0651c453003ff147fac125fc2c0471131fd0437ebc: Status 404 returned error can't find the container with id f6622ba5761556115ddcab0651c453003ff147fac125fc2c0471131fd0437ebc Oct 30 13:44:29.938588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:29.938546 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:30.157997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.157908 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" event={"ID":"df0d9328-129f-4911-97a0-27ed8e5b11f4","Type":"ContainerStarted","Data":"c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c"} Oct 30 13:44:30.157997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.157961 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" event={"ID":"df0d9328-129f-4911-97a0-27ed8e5b11f4","Type":"ContainerStarted","Data":"f6622ba5761556115ddcab0651c453003ff147fac125fc2c0471131fd0437ebc"} Oct 30 13:44:30.157997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.157987 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:30.163075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.163049 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln"] Oct 30 13:44:30.167874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.167857 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.194329 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.194302 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"agc-cred-acd4b1a0cabb\"" Oct 30 13:44:30.194587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.194448 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"kube-root-ca.crt\"" Oct 30 13:44:30.194587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.194559 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"payload-processing-external-model-e2e--fb32b92e-dockercfg-qkzmw\"" Oct 30 13:44:30.194790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.194649 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"agc-cred-62114b032ea8\"" Oct 30 13:44:30.194790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.194768 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"openshift-service-ca.crt\"" Oct 30 13:44:30.204449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.204423 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdws\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.204575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.204484 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.204927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.204904 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.205048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.205028 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.206639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.206572 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bd2fed\"" Oct 30 13:44:30.206776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.206666 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"routing-overlay\"" Oct 30 13:44:30.211429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.211401 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln"] Oct 30 13:44:30.222239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.222199 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" podStartSLOduration=1.222186846 podStartE2EDuration="1.222186846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:30.21779956 +0000 UTC m=+2446.084417733" watchObservedRunningTime="2026-10-30 13:44:30.222186846 +0000 UTC m=+2446.088805018" Oct 30 13:44:30.307308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.307270 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.307536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.307467 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.307763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.307730 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdws\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.307917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.307831 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.308076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.308053 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.308478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.308459 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.310295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.310275 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.355651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.355580 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdws\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.483710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.483658 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:30.855850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:30.855822 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln"] Oct 30 13:44:30.857236 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:30.857204 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78c8b83_98e3_4ef8_8620_22288b1a2c9f.slice/crio-35afc24524e9560923e93bf088c061f95088493572f9e79c932836584f7ff782 WatchSource:0}: Error finding container 35afc24524e9560923e93bf088c061f95088493572f9e79c932836584f7ff782: Status 404 returned error can't find the container with id 35afc24524e9560923e93bf088c061f95088493572f9e79c932836584f7ff782 Oct 30 13:44:31.235399 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:31.235363 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:31.240436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:31.240405 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" event={"ID":"f78c8b83-98e3-4ef8-8620-22288b1a2c9f","Type":"ContainerStarted","Data":"b5ea6d2c5eafdb35655d7b90ee39eade61434c5101fe01e10f30ffb6a0ba29e6"} Oct 30 13:44:31.240757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:31.240736 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" event={"ID":"f78c8b83-98e3-4ef8-8620-22288b1a2c9f","Type":"ContainerStarted","Data":"35afc24524e9560923e93bf088c061f95088493572f9e79c932836584f7ff782"} Oct 30 13:44:31.240931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:31.240909 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:31.330722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:31.330673 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" podStartSLOduration=1.330658833 podStartE2EDuration="1.330658833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:31.326778313 +0000 UTC m=+2447.193396489" watchObservedRunningTime="2026-10-30 13:44:31.330658833 +0000 UTC m=+2447.197277007" Oct 30 13:44:33.431133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:33.431094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:34.798622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:34.798123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:35.073678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:35.070451 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.103235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:36.103202 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.460386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:36.460303 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.504683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:36.500082 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-j6662"] Oct 30 13:44:36.504683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:36.503400 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:36.697641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:36.697592 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.982621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:36.982167 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:37.853683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:37.853646 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:38.234671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:38.233887 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:38.784120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:38.784084 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:39.129683 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:39.121380 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:40.917670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:40.917630 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:41.501014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.500969 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" podUID="df0d9328-129f-4911-97a0-27ed8e5b11f4" containerName="payload-processing" containerID="cri-o://c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c" gracePeriod=25 Oct 30 13:44:41.784918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.784892 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:41.908684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.908646 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4\") pod \"df0d9328-129f-4911-97a0-27ed8e5b11f4\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " Oct 30 13:44:41.908838 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.908703 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials\") pod \"df0d9328-129f-4911-97a0-27ed8e5b11f4\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " Oct 30 13:44:41.908988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.908968 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume\") pod \"df0d9328-129f-4911-97a0-27ed8e5b11f4\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " Oct 30 13:44:41.909080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.909062 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume\") pod \"df0d9328-129f-4911-97a0-27ed8e5b11f4\" (UID: \"df0d9328-129f-4911-97a0-27ed8e5b11f4\") " Oct 30 13:44:41.909411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.909383 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume" pod "df0d9328-129f-4911-97a0-27ed8e5b11f4" (UID: "df0d9328-129f-4911-97a0-27ed8e5b11f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:41.909630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.909575 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume" pod "df0d9328-129f-4911-97a0-27ed8e5b11f4" (UID: "df0d9328-129f-4911-97a0-27ed8e5b11f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:41.909813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.909793 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:41.909859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.909822 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0d9328-129f-4911-97a0-27ed8e5b11f4-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:41.910889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.910864 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4" pod "df0d9328-129f-4911-97a0-27ed8e5b11f4" (UID: "df0d9328-129f-4911-97a0-27ed8e5b11f4"). InnerVolumeSpecName "kube-api-access-mc8r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:41.910993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:41.910970 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials" pod "df0d9328-129f-4911-97a0-27ed8e5b11f4" (UID: "df0d9328-129f-4911-97a0-27ed8e5b11f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:42.010995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.010961 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8r4\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-kube-api-access-mc8r4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:42.010995 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.010990 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df0d9328-129f-4911-97a0-27ed8e5b11f4-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:44:42.612101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.612021 5693 generic.go:356] "Generic (PLEG): container finished" podID="df0d9328-129f-4911-97a0-27ed8e5b11f4" containerID="c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c" exitCode=0 Oct 30 13:44:42.612101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.612067 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" Oct 30 13:44:42.612296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.612106 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" event={"ID":"df0d9328-129f-4911-97a0-27ed8e5b11f4","Type":"ContainerDied","Data":"c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c"} Oct 30 13:44:42.612296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.612169 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b7d6584f7-j6662" event={"ID":"df0d9328-129f-4911-97a0-27ed8e5b11f4","Type":"ContainerDied","Data":"f6622ba5761556115ddcab0651c453003ff147fac125fc2c0471131fd0437ebc"} Oct 30 13:44:42.612296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.612195 5693 scope.go:122] "RemoveContainer" containerID="c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c" Oct 30 13:44:42.623388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.623370 5693 scope.go:122] "RemoveContainer" containerID="c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c" Oct 30 13:44:42.623676 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:42.623655 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c\": container with ID starting with c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c not found: ID does not exist" containerID="c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c" Oct 30 13:44:42.623739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.623684 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c"} err="failed to get container status \"c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c\": rpc error: code = NotFound desc = could not find container \"c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c\": container with ID starting with c7f7b911fcbedd33543d98c5bf6739054112966e660f809aa02ae8775c51d53c not found: ID does not exist" Oct 30 13:44:42.694777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.694739 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-j6662"] Oct 30 13:44:42.716420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.716388 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-b7d6584f7-j6662"] Oct 30 13:44:42.944810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:42.944720 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:43.419023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:43.418997 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:44:44.267858 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:44.267818 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0d9328-129f-4911-97a0-27ed8e5b11f4" path="/var/lib/kubelet/pods/df0d9328-129f-4911-97a0-27ed8e5b11f4/volumes" Oct 30 13:44:44.799396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:44.799359 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:46.597512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.596809 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:46.819054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.819018 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg"] Oct 30 13:44:46.820038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.820012 5693 cpu_manager.go:414] "containerMap: removing container" podUID="df0d9328-129f-4911-97a0-27ed8e5b11f4" containerName="payload-processing" Oct 30 13:44:46.820038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.820036 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0d9328-129f-4911-97a0-27ed8e5b11f4" containerName="payload-processing" Oct 30 13:44:46.820245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.820163 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0d9328-129f-4911-97a0-27ed8e5b11f4" containerName="payload-processing" Oct 30 13:44:46.825534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.825509 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.844175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.844150 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-74589c-openshift-default-dockercfg-2w2h6\"" Oct 30 13:44:46.859955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.859889 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg"] Oct 30 13:44:46.900499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900460 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.900708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900552 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.900708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900684 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.900799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.900869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.900913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900882 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.900982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.900961 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.901044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.901029 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:46.901195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:46.901178 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prg7h\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.002991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.002956 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prg7h\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003031 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003106 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003171 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003224 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003277 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003308 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003365 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003411 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003585 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003688 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003811 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003782 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003906 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.003975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.003955 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.005733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.005700 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.005929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.005910 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.032474 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.032446 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prg7h\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.032635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.032583 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.140379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.140290 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:47.473490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.473460 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg"] Oct 30 13:44:47.475965 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:44:47.475939 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd1fdd1_54e7_4d0e_bd6b_082d21021823.slice/crio-443db123519dfd64a720ebe531e19b2efa5309cb9a26481d2c9a1cabac8cb71c WatchSource:0}: Error finding container 443db123519dfd64a720ebe531e19b2efa5309cb9a26481d2c9a1cabac8cb71c: Status 404 returned error can't find the container with id 443db123519dfd64a720ebe531e19b2efa5309cb9a26481d2c9a1cabac8cb71c Oct 30 13:44:47.483272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.480092 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:47.483272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.480252 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:47.483272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.480339 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:47.653498 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.653452 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" event={"ID":"9cd1fdd1-54e7-4d0e-bd6b-082d21021823","Type":"ContainerStarted","Data":"4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d"} Oct 30 13:44:47.653950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.653561 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" event={"ID":"9cd1fdd1-54e7-4d0e-bd6b-082d21021823","Type":"ContainerStarted","Data":"443db123519dfd64a720ebe531e19b2efa5309cb9a26481d2c9a1cabac8cb71c"} Oct 30 13:44:47.714576 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.714520 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podStartSLOduration=1.714504351 podStartE2EDuration="1.714504351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:47.709890336 +0000 UTC m=+2463.576508510" watchObservedRunningTime="2026-10-30 13:44:47.714504351 +0000 UTC m=+2463.581122521" Oct 30 13:44:47.961735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:47.961701 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:48.143621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:48.142969 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:48.153292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:48.153258 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 13:44:48.153472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:48.153314 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 13:44:48.736622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:48.726222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.151890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:49.151786 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 13:44:49.151890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:49.151861 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 13:44:49.446080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:49.445995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.661038 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:49.661001 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:50.153659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:50.142952 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 13:44:50.153659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:50.143013 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 13:44:51.104492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:51.104408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:51.143629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:51.143461 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 13:44:51.143629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:51.143521 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 13:44:52.140975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:52.140937 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 13:44:52.141578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:52.141545 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 13:44:53.018706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:53.017713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:53.085841 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:44:53.085804 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:44:53.140998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:53.140957 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 13:44:53.141684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:53.141649 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 13:44:54.146352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:54.146322 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:54.146803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:54.146785 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:54.147593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:54.147577 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:44:54.869990 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:54.869955 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:56.409411 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:56.409379 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:56.786438 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:56.786403 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:58.525842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:44:58.523827 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:00.353310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.353278 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs"] Oct 30 13:45:00.362093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.362065 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.369220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.369171 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs"] Oct 30 13:45:00.376904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.376623 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb"] Oct 30 13:45:00.381368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.381337 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\"" Oct 30 13:45:00.381870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.381851 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.407649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.407561 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k"] Oct 30 13:45:00.415051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.414805 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb"] Oct 30 13:45:00.415503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.414900 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.415503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.415190 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:00.419874 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.419845 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz"] Oct 30 13:45:00.427708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.426437 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.440949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.440926 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j"] Oct 30 13:45:00.450425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.450402 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.456062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456026 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6m7c\" (UniqueName: \"kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.456204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456118 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.456204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456153 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtm9p\" (UniqueName: \"kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.456204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456191 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5zx\" (UniqueName: \"kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.456389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456265 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qpz\" (UniqueName: \"kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.456389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456338 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.456510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456442 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.456577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456511 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.456662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456580 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.456662 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.456634 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7zj\" (UniqueName: \"kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.463748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.463720 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k"] Oct 30 13:45:00.479620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.476320 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz"] Oct 30 13:45:00.501721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.498455 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j"] Oct 30 13:45:00.558099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558063 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.558289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558122 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.558289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558152 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7zj\" (UniqueName: \"kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.558289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558183 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6m7c\" (UniqueName: \"kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.558289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558242 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.558289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558273 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtm9p\" (UniqueName: \"kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.558581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5zx\" (UniqueName: \"kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.558581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558382 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qpz\" (UniqueName: \"kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.558581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558458 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.558581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558555 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.558961 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.558930 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.559122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.559091 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.559614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.559426 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.559723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.559677 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.559890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.559869 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.590007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.589967 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7zj\" (UniqueName: \"kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.590297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.590278 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtm9p\" (UniqueName: \"kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.590522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.590500 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qpz\" (UniqueName: \"kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.590608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.590561 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6m7c\" (UniqueName: \"kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.590792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.590774 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5zx\" (UniqueName: \"kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.683180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.683097 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:00.700729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.700707 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:00.748910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.748878 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:00.772982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.767590 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:00.803740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.803706 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:00.902100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.902059 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q"] Oct 30 13:45:00.910724 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.910698 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.935578 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.935509 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q"] Oct 30 13:45:00.966875 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.966755 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.966885 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967274 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967245 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967299 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967430 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967414 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hd8\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967532 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967638 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967690 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:00.967792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:00.967769 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.008333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.004174 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs"] Oct 30 13:45:01.025207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.023316 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb"] Oct 30 13:45:01.025207 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:45:01.023931 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa5fc1b_6e36_4910_8e95_0f71878f6cb1.slice/crio-8df02ac25c46244e965488df847449bb9bbbc4fb214c85a2278e2f9456f969f1 WatchSource:0}: Error finding container 8df02ac25c46244e965488df847449bb9bbbc4fb214c85a2278e2f9456f969f1: Status 404 returned error can't find the container with id 8df02ac25c46244e965488df847449bb9bbbc4fb214c85a2278e2f9456f969f1 Oct 30 13:45:01.069976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.069937 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070020 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070106 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070135 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070170 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070306 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070201 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070313 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hd8\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070439 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.070573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.070535 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.071102 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.071051 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.071224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.071141 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.071377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.071342 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.071684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.071644 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.072165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.072138 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.073372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.073349 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.073715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.073694 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.093587 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.093553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hd8\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.095800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.095770 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-v649q\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.126073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.126044 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz"] Oct 30 13:45:01.132040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.132013 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k"] Oct 30 13:45:01.133366 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:45:01.133342 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795cf0d1_0c49_42bd_b774_03f38280531d.slice/crio-ff21d584b3306c7bb098b245d17fd3a905131c20c1c7e73af6eb5a651c6a1362 WatchSource:0}: Error finding container ff21d584b3306c7bb098b245d17fd3a905131c20c1c7e73af6eb5a651c6a1362: Status 404 returned error can't find the container with id ff21d584b3306c7bb098b245d17fd3a905131c20c1c7e73af6eb5a651c6a1362 Oct 30 13:45:01.157350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.157317 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j"] Oct 30 13:45:01.231061 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:45:01.231018 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3938fe33_e533_40a0_a039_e3fe6a52cd1e.slice/crio-5a33c2dcb1d49b3f361220d9b763c5b50b598e7d2c53f5cf3f5800113b450507 WatchSource:0}: Error finding container 5a33c2dcb1d49b3f361220d9b763c5b50b598e7d2c53f5cf3f5800113b450507: Status 404 returned error can't find the container with id 5a33c2dcb1d49b3f361220d9b763c5b50b598e7d2c53f5cf3f5800113b450507 Oct 30 13:45:01.238847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.238821 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:01.560492 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:45:01.560457 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ec9ec5_eda4_4779_8d24_71f7dd7a7bfd.slice/crio-8e2dd037dd0f84e26c565514baf82ea02f16c1d7f1118dcc32fd307cf34e70d2 WatchSource:0}: Error finding container 8e2dd037dd0f84e26c565514baf82ea02f16c1d7f1118dcc32fd307cf34e70d2: Status 404 returned error can't find the container with id 8e2dd037dd0f84e26c565514baf82ea02f16c1d7f1118dcc32fd307cf34e70d2 Oct 30 13:45:01.561314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.561290 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q"] Oct 30 13:45:01.563572 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.563542 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:45:01.563652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.563626 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:45:01.563697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.563665 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:45:01.766197 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.766169 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" event={"ID":"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd","Type":"ContainerStarted","Data":"8e2dd037dd0f84e26c565514baf82ea02f16c1d7f1118dcc32fd307cf34e70d2"} Oct 30 13:45:01.768060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.768026 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerStarted","Data":"0a1cff149032b1020ebfe0e56f8f5adb345330b6761e302e426e25d23ce1d99e"} Oct 30 13:45:01.768185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.768093 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerStarted","Data":"8df02ac25c46244e965488df847449bb9bbbc4fb214c85a2278e2f9456f969f1"} Oct 30 13:45:01.770095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.770070 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerStarted","Data":"3dadcbaec5bc45275ebb83460e1dc6baa497aa65f3908f66afc085d37396af83"} Oct 30 13:45:01.770205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.770118 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerStarted","Data":"f1a35f1c9d213e6d861d21312f227267b8d5d03722bc28e8d774f8031246ad02"} Oct 30 13:45:01.771569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.771548 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerStarted","Data":"ff21d584b3306c7bb098b245d17fd3a905131c20c1c7e73af6eb5a651c6a1362"} Oct 30 13:45:01.773225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.773201 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerStarted","Data":"46189aa0e18f0bde6de6962f213c039697ee6c5a252078ed1efc405fe1a09c03"} Oct 30 13:45:01.773318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.773255 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerStarted","Data":"a563e48cf9afa3fb2079222bec9024a1effbe90fe27381f59ceeab30cb172d24"} Oct 30 13:45:01.774650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.774561 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" event={"ID":"3938fe33-e533-40a0-a039-e3fe6a52cd1e","Type":"ContainerStarted","Data":"5a33c2dcb1d49b3f361220d9b763c5b50b598e7d2c53f5cf3f5800113b450507"} Oct 30 13:45:01.816146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.816025 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" podStartSLOduration=1.816005934 podStartE2EDuration="1.816005934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.81291629 +0000 UTC m=+2477.679534464" watchObservedRunningTime="2026-10-30 13:45:01.816005934 +0000 UTC m=+2477.682624107" Oct 30 13:45:01.862807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.862744 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" podStartSLOduration=1.862722871 podStartE2EDuration="1.862722871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.861103371 +0000 UTC m=+2477.727721544" watchObservedRunningTime="2026-10-30 13:45:01.862722871 +0000 UTC m=+2477.729341045" Oct 30 13:45:01.906044 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:01.905991 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" podStartSLOduration=1.905977158 podStartE2EDuration="1.905977158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.901417633 +0000 UTC m=+2477.768035807" watchObservedRunningTime="2026-10-30 13:45:01.905977158 +0000 UTC m=+2477.772595343" Oct 30 13:45:02.273921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.273896 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp"] Oct 30 13:45:02.653716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.653540 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:02.785053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.784968 5693 generic.go:356] "Generic (PLEG): container finished" podID="795cf0d1-0c49-42bd-b774-03f38280531d" containerID="c3da9e55b1d909f1c1d00cef473ce3fa9e225c328c08d13823d7e677b4a96263" exitCode=60 Oct 30 13:45:02.785223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.785049 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerDied","Data":"c3da9e55b1d909f1c1d00cef473ce3fa9e225c328c08d13823d7e677b4a96263"} Oct 30 13:45:02.785420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.785399 5693 scope.go:122] "RemoveContainer" containerID="c3da9e55b1d909f1c1d00cef473ce3fa9e225c328c08d13823d7e677b4a96263" Oct 30 13:45:02.786923 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.786894 5693 generic.go:356] "Generic (PLEG): container finished" podID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerID="46189aa0e18f0bde6de6962f213c039697ee6c5a252078ed1efc405fe1a09c03" exitCode=60 Oct 30 13:45:02.787066 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.787000 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerDied","Data":"46189aa0e18f0bde6de6962f213c039697ee6c5a252078ed1efc405fe1a09c03"} Oct 30 13:45:02.787344 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.787322 5693 scope.go:122] "RemoveContainer" containerID="46189aa0e18f0bde6de6962f213c039697ee6c5a252078ed1efc405fe1a09c03" Oct 30 13:45:02.789568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.789544 5693 generic.go:356] "Generic (PLEG): container finished" podID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerID="5ccb4c59ee56ae6a1438554fc7b4ffef4bc1620fc1c7827015f3ed1717201bf0" exitCode=60 Oct 30 13:45:02.789729 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.789698 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" event={"ID":"3938fe33-e533-40a0-a039-e3fe6a52cd1e","Type":"ContainerDied","Data":"5ccb4c59ee56ae6a1438554fc7b4ffef4bc1620fc1c7827015f3ed1717201bf0"} Oct 30 13:45:02.789897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.789879 5693 scope.go:122] "RemoveContainer" containerID="5ccb4c59ee56ae6a1438554fc7b4ffef4bc1620fc1c7827015f3ed1717201bf0" Oct 30 13:45:02.792544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.792517 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" event={"ID":"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd","Type":"ContainerStarted","Data":"956bcd977a7cc64c055b341dbd898a604f1310aec7be8c22ec7c2c3030fb653c"} Oct 30 13:45:02.794925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.794897 5693 generic.go:356] "Generic (PLEG): container finished" podID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerID="0a1cff149032b1020ebfe0e56f8f5adb345330b6761e302e426e25d23ce1d99e" exitCode=60 Oct 30 13:45:02.795213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.795183 5693 scope.go:122] "RemoveContainer" containerID="0a1cff149032b1020ebfe0e56f8f5adb345330b6761e302e426e25d23ce1d99e" Oct 30 13:45:02.797403 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.797383 5693 generic.go:356] "Generic (PLEG): container finished" podID="ede01344-c35a-4f03-8850-065c68616c6b" containerID="3dadcbaec5bc45275ebb83460e1dc6baa497aa65f3908f66afc085d37396af83" exitCode=60 Oct 30 13:45:02.797735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.797713 5693 scope.go:122] "RemoveContainer" containerID="3dadcbaec5bc45275ebb83460e1dc6baa497aa65f3908f66afc085d37396af83" Oct 30 13:45:02.801774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.801729 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerDied","Data":"0a1cff149032b1020ebfe0e56f8f5adb345330b6761e302e426e25d23ce1d99e"} Oct 30 13:45:02.801920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.801795 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerDied","Data":"3dadcbaec5bc45275ebb83460e1dc6baa497aa65f3908f66afc085d37396af83"} Oct 30 13:45:02.968203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:02.968068 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.201514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.201457 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podStartSLOduration=3.201444084 podStartE2EDuration="3.201444084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.170447864 +0000 UTC m=+2479.037066039" watchObservedRunningTime="2026-10-30 13:45:03.201444084 +0000 UTC m=+2479.068062257" Oct 30 13:45:03.251622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.243803 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:03.251622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.243977 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:03.251622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.244032 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:03.321681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.317477 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.694382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.694347 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8"] Oct 30 13:45:03.699270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.699249 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.722105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.722076 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqgn\" (UniqueName: \"kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.722288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.722258 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.722401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.722384 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.768706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.766402 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:03.775149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.775123 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8"] Oct 30 13:45:03.817501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.817471 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerStarted","Data":"031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3"} Oct 30 13:45:03.822032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.822000 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerStarted","Data":"cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6"} Oct 30 13:45:03.823644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.823619 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgn\" (UniqueName: \"kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.823849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.823734 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.823849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.823803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.825119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.825041 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerStarted","Data":"4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99"} Oct 30 13:45:03.825374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.825353 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.861513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.861480 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:03.862110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.862080 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerStarted","Data":"53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4"} Oct 30 13:45:03.864919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.864890 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" event={"ID":"3938fe33-e533-40a0-a039-e3fe6a52cd1e","Type":"ContainerStarted","Data":"688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d"} Oct 30 13:45:03.867348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:03.867329 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqgn\" (UniqueName: \"kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn\") pod \"maas-api-revoke-sub-7385aabb-fwrv8\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:04.017476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.017437 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:04.064168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.064107 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" podStartSLOduration=4.064088132 podStartE2EDuration="4.064088132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.052948682 +0000 UTC m=+2479.919566859" watchObservedRunningTime="2026-10-30 13:45:04.064088132 +0000 UTC m=+2479.930706307" Oct 30 13:45:04.195983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.195915 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" podStartSLOduration=4.195896019 podStartE2EDuration="4.195896019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.195309208 +0000 UTC m=+2480.061927383" watchObservedRunningTime="2026-10-30 13:45:04.195896019 +0000 UTC m=+2480.062514194" Oct 30 13:45:04.251169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.249711 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:04.251169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.249773 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:04.376665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.376630 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8"] Oct 30 13:45:04.422483 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:45:04.422447 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24980b11_8a6c_407b_895c_696426008772.slice/crio-9560f716946045c26d5ac5f7dd957a561148ca5fee3f6966866f60ca822cf070 WatchSource:0}: Error finding container 9560f716946045c26d5ac5f7dd957a561148ca5fee3f6966866f60ca822cf070: Status 404 returned error can't find the container with id 9560f716946045c26d5ac5f7dd957a561148ca5fee3f6966866f60ca822cf070 Oct 30 13:45:04.876466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.876436 5693 generic.go:356] "Generic (PLEG): container finished" podID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerID="031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3" exitCode=60 Oct 30 13:45:04.877108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.876620 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerDied","Data":"031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3"} Oct 30 13:45:04.877335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.877098 5693 scope.go:122] "RemoveContainer" containerID="031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3" Oct 30 13:45:04.877548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.877528 5693 scope.go:122] "RemoveContainer" containerID="0a1cff149032b1020ebfe0e56f8f5adb345330b6761e302e426e25d23ce1d99e" Oct 30 13:45:04.877968 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:04.877945 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb_odh-ai-gateway-infra(0fa5fc1b-6e36-4910-8e95-0f71878f6cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" Oct 30 13:45:04.880532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.880512 5693 generic.go:356] "Generic (PLEG): container finished" podID="ede01344-c35a-4f03-8850-065c68616c6b" containerID="cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6" exitCode=60 Oct 30 13:45:04.880715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.880640 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerDied","Data":"cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6"} Oct 30 13:45:04.881049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.881034 5693 scope.go:122] "RemoveContainer" containerID="cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6" Oct 30 13:45:04.881435 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:04.881410 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs_odh-ai-gateway-infra(ede01344-c35a-4f03-8850-065c68616c6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" podUID="ede01344-c35a-4f03-8850-065c68616c6b" Oct 30 13:45:04.882254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.882236 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" event={"ID":"24980b11-8a6c-407b-895c-696426008772","Type":"ContainerStarted","Data":"9560f716946045c26d5ac5f7dd957a561148ca5fee3f6966866f60ca822cf070"} Oct 30 13:45:04.887190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.887158 5693 generic.go:356] "Generic (PLEG): container finished" podID="795cf0d1-0c49-42bd-b774-03f38280531d" containerID="4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99" exitCode=60 Oct 30 13:45:04.887286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.887213 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerDied","Data":"4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99"} Oct 30 13:45:04.887522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.887502 5693 scope.go:122] "RemoveContainer" containerID="4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99" Oct 30 13:45:04.887851 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:04.887831 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k_odh-ai-gateway-infra(795cf0d1-0c49-42bd-b774-03f38280531d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" Oct 30 13:45:04.889357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.889336 5693 generic.go:356] "Generic (PLEG): container finished" podID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerID="53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4" exitCode=60 Oct 30 13:45:04.889453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.889416 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerDied","Data":"53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4"} Oct 30 13:45:04.889771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.889755 5693 scope.go:122] "RemoveContainer" containerID="53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4" Oct 30 13:45:04.890066 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:04.890046 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz_odh-ai-gateway-infra(c1cafd9d-d290-48e1-98f6-879e09b69c93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" Oct 30 13:45:04.891513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.891490 5693 generic.go:356] "Generic (PLEG): container finished" podID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerID="688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d" exitCode=60 Oct 30 13:45:04.891627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.891548 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" event={"ID":"3938fe33-e533-40a0-a039-e3fe6a52cd1e","Type":"ContainerDied","Data":"688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d"} Oct 30 13:45:04.891828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.891814 5693 scope.go:122] "RemoveContainer" containerID="688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d" Oct 30 13:45:04.892074 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:04.892054 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j_odh-ai-gateway-infra(3938fe33-e533-40a0-a039-e3fe6a52cd1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" Oct 30 13:45:04.893137 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.893122 5693 scope.go:122] "RemoveContainer" containerID="3dadcbaec5bc45275ebb83460e1dc6baa497aa65f3908f66afc085d37396af83" Oct 30 13:45:04.908742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.908708 5693 scope.go:122] "RemoveContainer" containerID="c3da9e55b1d909f1c1d00cef473ce3fa9e225c328c08d13823d7e677b4a96263" Oct 30 13:45:04.927492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.927140 5693 scope.go:122] "RemoveContainer" containerID="46189aa0e18f0bde6de6962f213c039697ee6c5a252078ed1efc405fe1a09c03" Oct 30 13:45:04.970537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:04.970515 5693 scope.go:122] "RemoveContainer" containerID="5ccb4c59ee56ae6a1438554fc7b4ffef4bc1620fc1c7827015f3ed1717201bf0" Oct 30 13:45:05.255670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:05.252030 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:05.255670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:05.252089 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:05.463779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:05.452558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:06.041557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.033506 5693 scope.go:122] "RemoveContainer" containerID="031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3" Oct 30 13:45:06.041557 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:06.033963 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb_odh-ai-gateway-infra(0fa5fc1b-6e36-4910-8e95-0f71878f6cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" Oct 30 13:45:06.041557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.037714 5693 scope.go:122] "RemoveContainer" containerID="cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6" Oct 30 13:45:06.041557 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:06.038098 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs_odh-ai-gateway-infra(ede01344-c35a-4f03-8850-065c68616c6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" podUID="ede01344-c35a-4f03-8850-065c68616c6b" Oct 30 13:45:06.055517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.055481 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" event={"ID":"24980b11-8a6c-407b-895c-696426008772","Type":"ContainerStarted","Data":"6759e85bb4900672c39d6567690f14ce1705f0fc2fb84fd4cae252c8e4adab03"} Oct 30 13:45:06.059972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.059943 5693 scope.go:122] "RemoveContainer" containerID="4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99" Oct 30 13:45:06.060752 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:06.060722 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k_odh-ai-gateway-infra(795cf0d1-0c49-42bd-b774-03f38280531d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" Oct 30 13:45:06.067089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.066562 5693 scope.go:122] "RemoveContainer" containerID="53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4" Oct 30 13:45:06.067672 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:06.067647 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz_odh-ai-gateway-infra(c1cafd9d-d290-48e1-98f6-879e09b69c93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" Oct 30 13:45:06.072047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.072008 5693 scope.go:122] "RemoveContainer" containerID="688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d" Oct 30 13:45:06.073469 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:06.073445 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j_odh-ai-gateway-infra(3938fe33-e533-40a0-a039-e3fe6a52cd1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" Oct 30 13:45:06.249700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.246727 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:06.249700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.246784 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:06.512112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:06.510573 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" podStartSLOduration=3.510557682 podStartE2EDuration="3.510557682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:06.405149397 +0000 UTC m=+2482.271767570" watchObservedRunningTime="2026-10-30 13:45:06.510557682 +0000 UTC m=+2482.377175907" Oct 30 13:45:07.113626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:07.113197 5693 generic.go:356] "Generic (PLEG): container finished" podID="24980b11-8a6c-407b-895c-696426008772" containerID="6759e85bb4900672c39d6567690f14ce1705f0fc2fb84fd4cae252c8e4adab03" exitCode=0 Oct 30 13:45:07.113626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:07.113479 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" event={"ID":"24980b11-8a6c-407b-895c-696426008772","Type":"ContainerDied","Data":"6759e85bb4900672c39d6567690f14ce1705f0fc2fb84fd4cae252c8e4adab03"} Oct 30 13:45:07.255728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:07.255689 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:07.256000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:07.255973 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:07.282330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:07.282263 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" podUID="8471f82d-9ce6-4770-9729-44a7d430b482" containerName="payload-processing" containerID="cri-o://8ebe463bdb7859cd42561383369054b0d5ba2d5de223e04893de30cd8b01cff0" gracePeriod=25 Oct 30 13:45:08.324713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.318129 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:08.324713 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.318194 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:08.633029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.632940 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:08.706176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.706138 5693 generic.go:356] "Generic (PLEG): container finished" podID="8471f82d-9ce6-4770-9729-44a7d430b482" containerID="8ebe463bdb7859cd42561383369054b0d5ba2d5de223e04893de30cd8b01cff0" exitCode=0 Oct 30 13:45:08.737881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.707227 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" event={"ID":"8471f82d-9ce6-4770-9729-44a7d430b482","Type":"ContainerDied","Data":"8ebe463bdb7859cd42561383369054b0d5ba2d5de223e04893de30cd8b01cff0"} Oct 30 13:45:08.737881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.734252 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:45:08.844668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.843224 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials\") pod \"8471f82d-9ce6-4770-9729-44a7d430b482\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " Oct 30 13:45:08.844668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.843460 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume\") pod \"8471f82d-9ce6-4770-9729-44a7d430b482\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " Oct 30 13:45:08.844668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.843537 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume\") pod \"8471f82d-9ce6-4770-9729-44a7d430b482\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " Oct 30 13:45:08.844668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.843688 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9\") pod \"8471f82d-9ce6-4770-9729-44a7d430b482\" (UID: \"8471f82d-9ce6-4770-9729-44a7d430b482\") " Oct 30 13:45:08.848517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.848487 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume" pod "8471f82d-9ce6-4770-9729-44a7d430b482" (UID: "8471f82d-9ce6-4770-9729-44a7d430b482"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:08.856105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.856033 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume" pod "8471f82d-9ce6-4770-9729-44a7d430b482" (UID: "8471f82d-9ce6-4770-9729-44a7d430b482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:08.908475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.856422 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials" pod "8471f82d-9ce6-4770-9729-44a7d430b482" (UID: "8471f82d-9ce6-4770-9729-44a7d430b482"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:08.908475 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.904564 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9" pod "8471f82d-9ce6-4770-9729-44a7d430b482" (UID: "8471f82d-9ce6-4770-9729-44a7d430b482"). InnerVolumeSpecName "kube-api-access-rbcv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:08.955032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.953899 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.955032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.953950 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8471f82d-9ce6-4770-9729-44a7d430b482-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.955032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.953964 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbcv9\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-kube-api-access-rbcv9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.955032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:08.953980 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8471f82d-9ce6-4770-9729-44a7d430b482-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:09.083760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.083725 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:09.108651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.108615 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:09.157573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.157538 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn\" (UniqueName: \"kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn\") pod \"24980b11-8a6c-407b-895c-696426008772\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " Oct 30 13:45:09.157781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.157691 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca\") pod \"24980b11-8a6c-407b-895c-696426008772\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " Oct 30 13:45:09.157904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.157883 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca\") pod \"24980b11-8a6c-407b-895c-696426008772\" (UID: \"24980b11-8a6c-407b-895c-696426008772\") " Oct 30 13:45:09.158168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.158122 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca" pod "24980b11-8a6c-407b-895c-696426008772" (UID: "24980b11-8a6c-407b-895c-696426008772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:09.158739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.158678 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24980b11-8a6c-407b-895c-696426008772-maas-api-openshift-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:09.160225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.160200 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca" pod "24980b11-8a6c-407b-895c-696426008772" (UID: "24980b11-8a6c-407b-895c-696426008772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:09.161476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.160986 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn" pod "24980b11-8a6c-407b-895c-696426008772" (UID: "24980b11-8a6c-407b-895c-696426008772"). InnerVolumeSpecName "kube-api-access-vdqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:09.243664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.241199 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:09.243664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.241270 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:09.266992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.264700 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdqgn\" (UniqueName: \"kubernetes.io/projected/24980b11-8a6c-407b-895c-696426008772-kube-api-access-vdqgn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:09.266992 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.264737 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24980b11-8a6c-407b-895c-696426008772-maas-api-xks-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:09.329791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.329755 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:09.715708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.715678 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" Oct 30 13:45:09.715708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.715697 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp" event={"ID":"8471f82d-9ce6-4770-9729-44a7d430b482","Type":"ContainerDied","Data":"2ca43267e5d5ba2a69948a11ab1bcd7a591300b0766128836af4dac5cefd2438"} Oct 30 13:45:09.715966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.715789 5693 scope.go:122] "RemoveContainer" containerID="8ebe463bdb7859cd42561383369054b0d5ba2d5de223e04893de30cd8b01cff0" Oct 30 13:45:09.717396 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.717353 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" Oct 30 13:45:09.717512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.717374 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8" event={"ID":"24980b11-8a6c-407b-895c-696426008772","Type":"ContainerDied","Data":"9560f716946045c26d5ac5f7dd957a561148ca5fee3f6966866f60ca822cf070"} Oct 30 13:45:09.717512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.717497 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9560f716946045c26d5ac5f7dd957a561148ca5fee3f6966866f60ca822cf070" Oct 30 13:45:09.784291 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.784258 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp"] Oct 30 13:45:09.798676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:09.798640 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5b59gzfjp"] Oct 30 13:45:10.239620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:10.237997 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:10.257792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:10.248905 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v649q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 13:45:10.257792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:10.248981 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 13:45:10.336944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:10.336910 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8471f82d-9ce6-4770-9729-44a7d430b482" path="/var/lib/kubelet/pods/8471f82d-9ce6-4770-9729-44a7d430b482/volumes" Oct 30 13:45:11.450913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:11.450853 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:11.451505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:11.451484 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:11.452730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:11.452710 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:45:12.079620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:12.069708 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:13.977967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:13.977936 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.182539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:15.182499 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.247912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:15.247880 5693 scope.go:122] "RemoveContainer" containerID="4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99" Oct 30 13:45:15.424856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:15.424819 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.616032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:15.615992 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.800643 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:15.799494 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerStarted","Data":"1db8d757b2ef4464116d88f4ddb240a04d4ee922d3d14992667d33001e29ccc2"} Oct 30 13:45:16.248379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.248343 5693 scope.go:122] "RemoveContainer" containerID="688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d" Oct 30 13:45:16.265361 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.265327 5693 scope.go:122] "RemoveContainer" containerID="cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6" Oct 30 13:45:16.300621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.297507 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.586028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.585934 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.749029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.748995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.773949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.773880 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv"] Oct 30 13:45:16.775929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.775899 5693 cpu_manager.go:414] "containerMap: removing container" podUID="24980b11-8a6c-407b-895c-696426008772" containerName="revoke-keys" Oct 30 13:45:16.776122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.776106 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24980b11-8a6c-407b-895c-696426008772" containerName="revoke-keys" Oct 30 13:45:16.776334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.776320 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8471f82d-9ce6-4770-9729-44a7d430b482" containerName="payload-processing" Oct 30 13:45:16.776431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.776419 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8471f82d-9ce6-4770-9729-44a7d430b482" containerName="payload-processing" Oct 30 13:45:16.776757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.776738 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8471f82d-9ce6-4770-9729-44a7d430b482" containerName="payload-processing" Oct 30 13:45:16.776907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.776893 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="24980b11-8a6c-407b-895c-696426008772" containerName="revoke-keys" Oct 30 13:45:16.782320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.782298 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.786485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.786366 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv"] Oct 30 13:45:16.802758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.802732 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-587ab343f202\"" Oct 30 13:45:16.813417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.813381 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\"" Oct 30 13:45:16.814034 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.814016 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-v44rb\"" Oct 30 13:45:16.814390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.814374 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\"" Oct 30 13:45:16.814749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.814734 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\"" Oct 30 13:45:16.815097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.815081 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\"" Oct 30 13:45:16.833414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.833385 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.834119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.834082 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5k9\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.834400 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.834292 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.834557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.834531 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.835422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.835398 5693 generic.go:356] "Generic (PLEG): container finished" podID="795cf0d1-0c49-42bd-b774-03f38280531d" containerID="1db8d757b2ef4464116d88f4ddb240a04d4ee922d3d14992667d33001e29ccc2" exitCode=60 Oct 30 13:45:16.835830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.835766 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerDied","Data":"1db8d757b2ef4464116d88f4ddb240a04d4ee922d3d14992667d33001e29ccc2"} Oct 30 13:45:16.836093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.836044 5693 scope.go:122] "RemoveContainer" containerID="4864c2c5fff2d8f125fa03187b06d577353512831400bbde3b66ad69c949fa99" Oct 30 13:45:16.836782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.836767 5693 scope.go:122] "RemoveContainer" containerID="1db8d757b2ef4464116d88f4ddb240a04d4ee922d3d14992667d33001e29ccc2" Oct 30 13:45:16.837460 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:16.837440 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k_odh-ai-gateway-infra(795cf0d1-0c49-42bd-b774-03f38280531d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" Oct 30 13:45:16.868707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.868674 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k"] Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.942818 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.943052 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5k9\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.943126 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.943157 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:16.943885 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:16.943902 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv: secret "agc-cred-587ab343f202" not found Oct 30 13:45:16.944620 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:16.943970 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials podName:c6c44034-b890-48e4-936e-adccc5a36779 nodeName:}" failed. No retries permitted until 2026-10-30 13:45:17.443948911 +0000 UTC m=+2493.310567080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" (UID: "c6c44034-b890-48e4-936e-adccc5a36779") : secret "agc-cred-587ab343f202" not found Oct 30 13:45:17.030645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.030271 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:17.030645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:16.945588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:17.138630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.138297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:17.146675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.139279 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5k9\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:17.213903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.213866 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv"] Oct 30 13:45:17.214275 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:17.214246 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" podUID="c6c44034-b890-48e4-936e-adccc5a36779" Oct 30 13:45:17.247878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.247845 5693 scope.go:122] "RemoveContainer" containerID="031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3" Oct 30 13:45:17.352363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.352256 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:17.514251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.514212 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:17.514442 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:17.514361 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found Oct 30 13:45:17.514442 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:17.514382 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv: secret "agc-cred-587ab343f202" not found Oct 30 13:45:17.514442 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:17.514432 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials podName:c6c44034-b890-48e4-936e-adccc5a36779 nodeName:}" failed. No retries permitted until 2026-10-30 13:45:18.514415892 +0000 UTC m=+2494.381034044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" (UID: "c6c44034-b890-48e4-936e-adccc5a36779") : secret "agc-cred-587ab343f202" not found Oct 30 13:45:17.519114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.519085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:17.845677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.845643 5693 generic.go:356] "Generic (PLEG): container finished" podID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerID="b05958509d787ae566d2abd2e9bd3b73c7ff32aeef0ffe98869970dc817779c2" exitCode=60 Oct 30 13:45:17.845868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.845712 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" event={"ID":"3938fe33-e533-40a0-a039-e3fe6a52cd1e","Type":"ContainerDied","Data":"b05958509d787ae566d2abd2e9bd3b73c7ff32aeef0ffe98869970dc817779c2"} Oct 30 13:45:17.845868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.845773 5693 scope.go:122] "RemoveContainer" containerID="688b64489739cc94aa6222e6a6b41743f7f3ec8a3884a8eee958a9a4a27d801d" Oct 30 13:45:17.846239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.846221 5693 scope.go:122] "RemoveContainer" containerID="b05958509d787ae566d2abd2e9bd3b73c7ff32aeef0ffe98869970dc817779c2" Oct 30 13:45:17.846637 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:17.846615 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j_odh-ai-gateway-infra(3938fe33-e533-40a0-a039-e3fe6a52cd1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" Oct 30 13:45:17.848027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.848003 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerStarted","Data":"ba582988a679a0bce0836604bd4129d706fcd058a70ce65e197781fe8ded0b8b"} Oct 30 13:45:17.849939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.849915 5693 generic.go:356] "Generic (PLEG): container finished" podID="ede01344-c35a-4f03-8850-065c68616c6b" containerID="39f526a348d2ae01734267b1b6d4ced27f0f0ae25637748c358afff014785cb1" exitCode=60 Oct 30 13:45:17.850070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.849982 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerDied","Data":"39f526a348d2ae01734267b1b6d4ced27f0f0ae25637748c358afff014785cb1"} Oct 30 13:45:17.850472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.850443 5693 scope.go:122] "RemoveContainer" containerID="39f526a348d2ae01734267b1b6d4ced27f0f0ae25637748c358afff014785cb1" Oct 30 13:45:17.850815 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:17.850791 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs_odh-ai-gateway-infra(ede01344-c35a-4f03-8850-065c68616c6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" podUID="ede01344-c35a-4f03-8850-065c68616c6b" Oct 30 13:45:17.851848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.851831 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:17.860462 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.860441 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:17.863502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.863481 5693 scope.go:122] "RemoveContainer" containerID="cc05d419fdc61bf526920408f1dfe7f14870162c45052ce89686ad7f8fcbb6b6" Oct 30 13:45:17.923054 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.923021 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume\") pod \"c6c44034-b890-48e4-936e-adccc5a36779\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " Oct 30 13:45:17.923217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.923147 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume\") pod \"c6c44034-b890-48e4-936e-adccc5a36779\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " Oct 30 13:45:17.923421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.923401 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9\") pod \"c6c44034-b890-48e4-936e-adccc5a36779\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " Oct 30 13:45:17.923644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.923565 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume" pod "c6c44034-b890-48e4-936e-adccc5a36779" (UID: "c6c44034-b890-48e4-936e-adccc5a36779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:17.924139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.924097 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume" pod "c6c44034-b890-48e4-936e-adccc5a36779" (UID: "c6c44034-b890-48e4-936e-adccc5a36779"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:17.924289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.924262 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.924386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.924291 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c44034-b890-48e4-936e-adccc5a36779-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.932045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:17.931968 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9" pod "c6c44034-b890-48e4-936e-adccc5a36779" (UID: "c6c44034-b890-48e4-936e-adccc5a36779"). InnerVolumeSpecName "kube-api-access-hb5k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:18.025467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.025432 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5k9\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-kube-api-access-hb5k9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:18.056377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.056257 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:18.126008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.125969 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca\") pod \"795cf0d1-0c49-42bd-b774-03f38280531d\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " Oct 30 13:45:18.126347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.126325 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p\" (UniqueName: \"kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p\") pod \"795cf0d1-0c49-42bd-b774-03f38280531d\" (UID: \"795cf0d1-0c49-42bd-b774-03f38280531d\") " Oct 30 13:45:18.126447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.126417 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca" pod "795cf0d1-0c49-42bd-b774-03f38280531d" (UID: "795cf0d1-0c49-42bd-b774-03f38280531d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:18.127185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.127152 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795cf0d1-0c49-42bd-b774-03f38280531d-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:18.128532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.128509 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p" pod "795cf0d1-0c49-42bd-b774-03f38280531d" (UID: "795cf0d1-0c49-42bd-b774-03f38280531d"). InnerVolumeSpecName "kube-api-access-xtm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:18.228861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.228822 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtm9p\" (UniqueName: \"kubernetes.io/projected/795cf0d1-0c49-42bd-b774-03f38280531d-kube-api-access-xtm9p\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:18.258085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.258055 5693 scope.go:122] "RemoveContainer" containerID="53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4" Oct 30 13:45:18.529705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.529539 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:18.534583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.534543 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv\" (UID: \"c6c44034-b890-48e4-936e-adccc5a36779\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:18.561082 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:18.534927 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found Oct 30 13:45:18.561373 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:18.561351 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv: secret "agc-cred-587ab343f202" not found Oct 30 13:45:18.561576 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:18.561563 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials podName:c6c44034-b890-48e4-936e-adccc5a36779 nodeName:}" failed. No retries permitted until 2026-10-30 13:45:20.561537277 +0000 UTC m=+2496.428155440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" (UID: "c6c44034-b890-48e4-936e-adccc5a36779") : secret "agc-cred-587ab343f202" not found Oct 30 13:45:18.725806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.725772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:18.863575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.863478 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" event={"ID":"795cf0d1-0c49-42bd-b774-03f38280531d","Type":"ContainerDied","Data":"ff21d584b3306c7bb098b245d17fd3a905131c20c1c7e73af6eb5a651c6a1362"} Oct 30 13:45:18.863766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.863492 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k" Oct 30 13:45:18.863766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.863638 5693 scope.go:122] "RemoveContainer" containerID="1db8d757b2ef4464116d88f4ddb240a04d4ee922d3d14992667d33001e29ccc2" Oct 30 13:45:18.865825 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.865795 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerStarted","Data":"454d93d7cda15d8419f4770171a196c2d97a389e103753bc14db43d75e7fc9bb"} Oct 30 13:45:18.870527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.870501 5693 generic.go:356] "Generic (PLEG): container finished" podID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerID="ba582988a679a0bce0836604bd4129d706fcd058a70ce65e197781fe8ded0b8b" exitCode=60 Oct 30 13:45:18.870652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.870564 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerDied","Data":"ba582988a679a0bce0836604bd4129d706fcd058a70ce65e197781fe8ded0b8b"} Oct 30 13:45:18.870706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.870657 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv" Oct 30 13:45:18.870892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.870878 5693 scope.go:122] "RemoveContainer" containerID="ba582988a679a0bce0836604bd4129d706fcd058a70ce65e197781fe8ded0b8b" Oct 30 13:45:18.871147 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:18.871129 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb_odh-ai-gateway-infra(0fa5fc1b-6e36-4910-8e95-0f71878f6cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" Oct 30 13:45:18.876468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.876450 5693 scope.go:122] "RemoveContainer" containerID="031f5210472cabc1968aff5770c24fe9cb0d41f1f4ebc1462f937552c694afe3" Oct 30 13:45:18.906271 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.906247 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j"] Oct 30 13:45:18.957486 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:18.957452 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs"] Oct 30 13:45:19.014523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.014486 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k"] Oct 30 13:45:19.028043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.027493 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889465-q5t2k"] Oct 30 13:45:19.034655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.032923 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb"] Oct 30 13:45:19.111998 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.111275 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv"] Oct 30 13:45:19.118644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.118544 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775dfnjcv"] Oct 30 13:45:19.171048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.171019 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:19.179654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.179578 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c44034-b890-48e4-936e-adccc5a36779-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.206885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.206859 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:19.281712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.281679 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz\" (UniqueName: \"kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz\") pod \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " Oct 30 13:45:19.281912 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.281769 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca\") pod \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\" (UID: \"3938fe33-e533-40a0-a039-e3fe6a52cd1e\") " Oct 30 13:45:19.282005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.281978 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c\" (UniqueName: \"kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c\") pod \"ede01344-c35a-4f03-8850-065c68616c6b\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " Oct 30 13:45:19.282074 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.282008 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca\") pod \"ede01344-c35a-4f03-8850-065c68616c6b\" (UID: \"ede01344-c35a-4f03-8850-065c68616c6b\") " Oct 30 13:45:19.283969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.283937 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca" pod "3938fe33-e533-40a0-a039-e3fe6a52cd1e" (UID: "3938fe33-e533-40a0-a039-e3fe6a52cd1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:19.284127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.284108 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca" pod "ede01344-c35a-4f03-8850-065c68616c6b" (UID: "ede01344-c35a-4f03-8850-065c68616c6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:19.285425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.285394 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c" pod "ede01344-c35a-4f03-8850-065c68616c6b" (UID: "ede01344-c35a-4f03-8850-065c68616c6b"). InnerVolumeSpecName "kube-api-access-g6m7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:19.285917 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.285837 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz" pod "3938fe33-e533-40a0-a039-e3fe6a52cd1e" (UID: "3938fe33-e533-40a0-a039-e3fe6a52cd1e"). InnerVolumeSpecName "kube-api-access-b7qpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:19.384363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.384263 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qpz\" (UniqueName: \"kubernetes.io/projected/3938fe33-e533-40a0-a039-e3fe6a52cd1e-kube-api-access-b7qpz\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.384363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.384303 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3938fe33-e533-40a0-a039-e3fe6a52cd1e-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.384363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.384317 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6m7c\" (UniqueName: \"kubernetes.io/projected/ede01344-c35a-4f03-8850-065c68616c6b-kube-api-access-g6m7c\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.384363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.384330 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede01344-c35a-4f03-8850-065c68616c6b-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.881848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.881822 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" Oct 30 13:45:19.882222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.881865 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs" event={"ID":"ede01344-c35a-4f03-8850-065c68616c6b","Type":"ContainerDied","Data":"f1a35f1c9d213e6d861d21312f227267b8d5d03722bc28e8d774f8031246ad02"} Oct 30 13:45:19.882222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.881930 5693 scope.go:122] "RemoveContainer" containerID="39f526a348d2ae01734267b1b6d4ced27f0f0ae25637748c358afff014785cb1" Oct 30 13:45:19.885663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.885635 5693 generic.go:356] "Generic (PLEG): container finished" podID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerID="454d93d7cda15d8419f4770171a196c2d97a389e103753bc14db43d75e7fc9bb" exitCode=60 Oct 30 13:45:19.885800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.885729 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerDied","Data":"454d93d7cda15d8419f4770171a196c2d97a389e103753bc14db43d75e7fc9bb"} Oct 30 13:45:19.886189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.886169 5693 scope.go:122] "RemoveContainer" containerID="454d93d7cda15d8419f4770171a196c2d97a389e103753bc14db43d75e7fc9bb" Oct 30 13:45:19.886584 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:19.886561 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz_odh-ai-gateway-infra(c1cafd9d-d290-48e1-98f6-879e09b69c93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" Oct 30 13:45:19.888433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.888406 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" event={"ID":"3938fe33-e533-40a0-a039-e3fe6a52cd1e","Type":"ContainerDied","Data":"5a33c2dcb1d49b3f361220d9b763c5b50b598e7d2c53f5cf3f5800113b450507"} Oct 30 13:45:19.888548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.888482 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j" Oct 30 13:45:19.894159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.894140 5693 scope.go:122] "RemoveContainer" containerID="53989184451d4dc934d9b276ca1d3efc4cb0c92567f58e2ad1ba145aa2f414b4" Oct 30 13:45:19.911039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:19.909213 5693 scope.go:122] "RemoveContainer" containerID="b05958509d787ae566d2abd2e9bd3b73c7ff32aeef0ffe98869970dc817779c2" Oct 30 13:45:20.023819 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.023789 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j"] Oct 30 13:45:20.030350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.030318 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889465-9qc9j"] Oct 30 13:45:20.057272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.057238 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz"] Oct 30 13:45:20.062671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.062644 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs"] Oct 30 13:45:20.070881 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.070853 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889465-8fwjs"] Oct 30 13:45:20.084321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.084300 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:20.194384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.194348 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca\") pod \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " Oct 30 13:45:20.194692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.194593 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx\" (UniqueName: \"kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx\") pod \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\" (UID: \"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1\") " Oct 30 13:45:20.194829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.194798 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca" pod "0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" (UID: "0fa5fc1b-6e36-4910-8e95-0f71878f6cb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.195303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.195259 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.197293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.197267 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx" pod "0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" (UID: "0fa5fc1b-6e36-4910-8e95-0f71878f6cb1"). InnerVolumeSpecName "kube-api-access-mf5zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.255988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.255957 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" path="/var/lib/kubelet/pods/3938fe33-e533-40a0-a039-e3fe6a52cd1e/volumes" Oct 30 13:45:20.256634 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.256593 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" path="/var/lib/kubelet/pods/795cf0d1-0c49-42bd-b774-03f38280531d/volumes" Oct 30 13:45:20.257159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.257139 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c44034-b890-48e4-936e-adccc5a36779" path="/var/lib/kubelet/pods/c6c44034-b890-48e4-936e-adccc5a36779/volumes" Oct 30 13:45:20.257549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.257531 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede01344-c35a-4f03-8850-065c68616c6b" path="/var/lib/kubelet/pods/ede01344-c35a-4f03-8850-065c68616c6b/volumes" Oct 30 13:45:20.297208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.297178 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5zx\" (UniqueName: \"kubernetes.io/projected/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1-kube-api-access-mf5zx\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.536621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.534238 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:20.901400 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.901373 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" Oct 30 13:45:20.901837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.901441 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb" event={"ID":"0fa5fc1b-6e36-4910-8e95-0f71878f6cb1","Type":"ContainerDied","Data":"8df02ac25c46244e965488df847449bb9bbbc4fb214c85a2278e2f9456f969f1"} Oct 30 13:45:20.901837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.901503 5693 scope.go:122] "RemoveContainer" containerID="ba582988a679a0bce0836604bd4129d706fcd058a70ce65e197781fe8ded0b8b" Oct 30 13:45:20.981318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.981285 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb"] Oct 30 13:45:20.994561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:20.994529 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889465-m74fb"] Oct 30 13:45:21.072509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.072482 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:21.216024 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.215936 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca\") pod \"c1cafd9d-d290-48e1-98f6-879e09b69c93\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " Oct 30 13:45:21.216199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.216055 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj\" (UniqueName: \"kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj\") pod \"c1cafd9d-d290-48e1-98f6-879e09b69c93\" (UID: \"c1cafd9d-d290-48e1-98f6-879e09b69c93\") " Oct 30 13:45:21.216304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.216277 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca" pod "c1cafd9d-d290-48e1-98f6-879e09b69c93" (UID: "c1cafd9d-d290-48e1-98f6-879e09b69c93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:21.216668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.216644 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cafd9d-d290-48e1-98f6-879e09b69c93-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.218343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.218321 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj" pod "c1cafd9d-d290-48e1-98f6-879e09b69c93" (UID: "c1cafd9d-d290-48e1-98f6-879e09b69c93"). InnerVolumeSpecName "kube-api-access-vd7zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:21.318521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.318486 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7zj\" (UniqueName: \"kubernetes.io/projected/c1cafd9d-d290-48e1-98f6-879e09b69c93-kube-api-access-vd7zj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.913439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.913408 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" Oct 30 13:45:21.913958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.913436 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz" event={"ID":"c1cafd9d-d290-48e1-98f6-879e09b69c93","Type":"ContainerDied","Data":"a563e48cf9afa3fb2079222bec9024a1effbe90fe27381f59ceeab30cb172d24"} Oct 30 13:45:21.913958 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.913507 5693 scope.go:122] "RemoveContainer" containerID="454d93d7cda15d8419f4770171a196c2d97a389e103753bc14db43d75e7fc9bb" Oct 30 13:45:21.981682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.981654 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz"] Oct 30 13:45:21.987896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:21.987872 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889465-cz8sz"] Oct 30 13:45:22.260099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:22.260066 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" path="/var/lib/kubelet/pods/0fa5fc1b-6e36-4910-8e95-0f71878f6cb1/volumes" Oct 30 13:45:22.260766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:22.260743 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" path="/var/lib/kubelet/pods/c1cafd9d-d290-48e1-98f6-879e09b69c93/volumes" Oct 30 13:45:22.456700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:22.453967 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:23.061112 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:23.061068 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:45:24.329623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:24.321405 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:25.832894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:25.832854 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:26.058152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:26.058109 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:26.279134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:26.279104 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:28.351110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:28.351073 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:30.118782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:30.118749 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.904579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:31.904342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:33.523804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:33.523749 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:33.712159 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:33.712115 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:33.913679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:33.908940 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:34.095783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:34.095751 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:34.368422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:34.368383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:35.108120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:35.108088 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:35.289297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:35.289262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:36.033219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:36.033186 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:36.903623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:36.897762 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:37.893040 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:37.893001 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:38.118756 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.118717 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66"] Oct 30 13:45:38.119684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119652 5693 cpu_manager.go:414] "containerMap: removing container" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:45:38.119684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119677 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119708 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119714 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119725 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119731 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119751 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119755 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119767 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119772 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119779 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119785 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119806 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119823 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119832 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119837 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.119834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119843 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119847 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119856 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119861 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119875 5693 cpu_manager.go:414] "containerMap: removing container" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119879 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119895 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.119899 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120048 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120068 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120078 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120088 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120094 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120103 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120108 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120117 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120122 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa5fc1b-6e36-4910-8e95-0f71878f6cb1" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120127 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:45:38.120669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.120133 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:45:38.126078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.126043 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.141792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.141764 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66"] Oct 30 13:45:38.151679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.149474 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"routing-overlay\"" Oct 30 13:45:38.151679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.149488 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-98f1cc\"" Oct 30 13:45:38.151679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.149505 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"kube-root-ca.crt\"" Oct 30 13:45:38.151679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.149554 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"openshift-service-ca.crt\"" Oct 30 13:45:38.151679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.149583 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"agc-cred-e539133cb1f7\"" Oct 30 13:45:38.152583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.152544 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.152716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.152648 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqlj\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.152716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.152705 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.152843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.152769 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.154788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.154770 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"payload-processing-external-model-e2e--8b865544-dockercfg-rbmwm\"" Oct 30 13:45:38.276624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.274107 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.276624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.274199 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqlj\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.276624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.274291 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.276624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.274388 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.276624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.275370 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.318624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.300411 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.348941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.345267 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqlj\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.367220 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.367183 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.438753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.428234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:38.445081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.444975 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:38.805903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:38.805860 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66"] Oct 30 13:45:38.808278 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:45:38.808243 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7893b146_058a_40dc_963c_9d2726ffb0f8.slice/crio-e858c90e7c3195b393085df1b10633316f2204b8da9bc385a2ad3be142df65b8 WatchSource:0}: Error finding container e858c90e7c3195b393085df1b10633316f2204b8da9bc385a2ad3be142df65b8: Status 404 returned error can't find the container with id e858c90e7c3195b393085df1b10633316f2204b8da9bc385a2ad3be142df65b8 Oct 30 13:45:39.096039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:39.095939 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" event={"ID":"7893b146-058a-40dc-963c-9d2726ffb0f8","Type":"ContainerStarted","Data":"3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d"} Oct 30 13:45:39.096039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:39.096036 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" event={"ID":"7893b146-058a-40dc-963c-9d2726ffb0f8","Type":"ContainerStarted","Data":"e858c90e7c3195b393085df1b10633316f2204b8da9bc385a2ad3be142df65b8"} Oct 30 13:45:39.096256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:39.096144 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:40.277661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:40.272542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:42.438149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:42.438117 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:44.269655 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:44.269000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:46.053621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:46.051145 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:47.978620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:47.977826 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:49.734622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:49.725024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:50.108698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:50.107593 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:45:50.148293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:50.148240 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" podStartSLOduration=12.148221049 podStartE2EDuration="12.148221049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:39.16182095 +0000 UTC m=+2515.028439123" watchObservedRunningTime="2026-10-30 13:45:50.148221049 +0000 UTC m=+2526.014839223" Oct 30 13:45:51.585160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:51.585118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:53.057013 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:45:53.056974 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:45:53.407695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:53.407570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:55.203815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:55.203721 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:56.294296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:56.294262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:56.915109 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:56.915068 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:58.539278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:45:58.539237 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:00.321817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:00.321777 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:00.998308 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:00.998270 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:01.141765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:01.141727 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:01.620935 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:01.620900 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:02.135450 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:02.135381 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:04.034749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:04.034705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:05.374266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:05.374231 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:05.510096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:05.510062 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:05.722681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:05.721031 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:06.657629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:06.656158 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:07.724865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:07.724830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:08.891493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:08.891456 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:09.043276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:09.043239 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:09.188656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:09.188548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:09.383791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:09.383747 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:11.113390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:11.113353 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:12.854638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:12.841215 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:14.759016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:14.758978 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:16.557859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:16.557816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:16.713639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:16.713593 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:17.603921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:17.603888 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:17.793621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:17.793233 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:18.865999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:18.864467 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:19.076105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:19.076067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:20.750184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:20.750108 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:22.403626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:22.403046 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:23.056656 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:46:23.056617 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:46:24.120212 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:24.120174 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:24.303589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:24.303550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:24.452903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:24.452820 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:25.917023 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:25.916985 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:27.273569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:27.273520 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:27.472259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:27.472226 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:27.680319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:27.680171 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.049138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:28.049105 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.278645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:28.278606 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.658800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:28.658754 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.848622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:28.843228 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:29.991119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:29.991086 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:30.817443 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:30.817409 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:31.672482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:31.672444 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:31.844618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:31.843803 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:33.592701 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:33.592669 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:35.492620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:35.484489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:37.490624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:37.478499 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:39.688882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:39.688804 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:41.474346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:41.472872 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:43.015236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:43.015203 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:44.728801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:44.728758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:46.444063 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:46.444018 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:47.970855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:47.970815 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:49.615176 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:49.615142 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:50.162873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:50.162839 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:51.423152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:51.423121 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:53.057927 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:46:53.057885 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:46:53.105666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:53.105635 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:54.899787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:54.899748 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:56.412180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:56.412145 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:56.576270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:56.576229 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:58.159988 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:58.159953 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:59.801276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:46:59.801245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:01.512622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:01.508430 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:01.687791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:01.687756 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:01.858941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:01.858537 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:02.060338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:02.060305 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:02.231311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:02.231277 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:02.903100 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:02.903061 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.061485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:03.061449 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.278978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:03.278917 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.431854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:03.431823 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.575166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:03.575036 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.722666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:03.722631 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.938517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:03.938427 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:04.190082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:04.189984 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln"] Oct 30 13:47:04.297234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:04.297201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:05.506681 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:05.498248 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:07.163757 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:07.163722 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:09.190408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.190368 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" podUID="f78c8b83-98e3-4ef8-8620-22288b1a2c9f" containerName="payload-processing" containerID="cri-o://b5ea6d2c5eafdb35655d7b90ee39eade61434c5101fe01e10f30ffb6a0ba29e6" gracePeriod=25 Oct 30 13:47:09.299294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.299236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:09.380720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.380675 5693 generic.go:356] "Generic (PLEG): container finished" podID="f78c8b83-98e3-4ef8-8620-22288b1a2c9f" containerID="b5ea6d2c5eafdb35655d7b90ee39eade61434c5101fe01e10f30ffb6a0ba29e6" exitCode=0 Oct 30 13:47:09.380720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.380719 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" event={"ID":"f78c8b83-98e3-4ef8-8620-22288b1a2c9f","Type":"ContainerDied","Data":"b5ea6d2c5eafdb35655d7b90ee39eade61434c5101fe01e10f30ffb6a0ba29e6"} Oct 30 13:47:09.527332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.527302 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:47:09.673786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.673757 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume\") pod \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " Oct 30 13:47:09.673957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.673802 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials\") pod \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " Oct 30 13:47:09.674048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.673964 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws\") pod \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " Oct 30 13:47:09.674115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.674057 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume\") pod \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\" (UID: \"f78c8b83-98e3-4ef8-8620-22288b1a2c9f\") " Oct 30 13:47:09.674115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.674074 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume" pod "f78c8b83-98e3-4ef8-8620-22288b1a2c9f" (UID: "f78c8b83-98e3-4ef8-8620-22288b1a2c9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:47:09.674503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.674475 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume" pod "f78c8b83-98e3-4ef8-8620-22288b1a2c9f" (UID: "f78c8b83-98e3-4ef8-8620-22288b1a2c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:47:09.674794 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.674779 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:47:09.674856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.674797 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:47:09.676160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.676137 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws" pod "f78c8b83-98e3-4ef8-8620-22288b1a2c9f" (UID: "f78c8b83-98e3-4ef8-8620-22288b1a2c9f"). InnerVolumeSpecName "kube-api-access-6sdws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:47:09.676246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.676172 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials" pod "f78c8b83-98e3-4ef8-8620-22288b1a2c9f" (UID: "f78c8b83-98e3-4ef8-8620-22288b1a2c9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:47:09.777064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.776967 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdws\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-kube-api-access-6sdws\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:47:09.777064 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:09.777004 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f78c8b83-98e3-4ef8-8620-22288b1a2c9f-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:47:10.389585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.389542 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" event={"ID":"f78c8b83-98e3-4ef8-8620-22288b1a2c9f","Type":"ContainerDied","Data":"35afc24524e9560923e93bf088c061f95088493572f9e79c932836584f7ff782"} Oct 30 13:47:10.389585 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.389582 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln" Oct 30 13:47:10.390177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.389635 5693 scope.go:122] "RemoveContainer" containerID="b5ea6d2c5eafdb35655d7b90ee39eade61434c5101fe01e10f30ffb6a0ba29e6" Oct 30 13:47:10.441632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.441583 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln"] Oct 30 13:47:10.452712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.452678 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6d9984jln"] Oct 30 13:47:10.805879 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.805843 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:10.951080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:10.951047 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:11.091217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:11.091122 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:11.635710 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:11.635675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:11.783369 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:11.783323 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:11.938139 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:11.938056 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:12.071039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:12.071000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:12.256288 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:12.256253 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78c8b83-98e3-4ef8-8620-22288b1a2c9f" path="/var/lib/kubelet/pods/f78c8b83-98e3-4ef8-8620-22288b1a2c9f/volumes" Oct 30 13:47:12.714320 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:12.714222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:14.431304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:14.431236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:16.080623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:16.080252 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:17.760621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:17.760570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:18.313254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:18.313220 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:18.457867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:18.457832 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:19.056513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:19.054652 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:19.219842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:19.219807 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:19.394480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:19.394401 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:19.551236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:19.551201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:19.712539 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:19.712411 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:20.442762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:20.442729 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:20.593609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:20.593555 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:20.776245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:20.776202 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:21.721762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:21.721717 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:21.986094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:21.986054 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:23.058762 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:47:23.058727 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:47:23.766205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:23.766154 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:25.428830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:25.428748 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:27.243232 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:27.243198 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:29.008680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:29.008647 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:30.714631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:30.700532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:32.254321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:32.254274 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:33.868661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:33.863844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:35.531463 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:35.531259 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:37.322523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:37.322489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:38.994293 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:38.994248 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:40.595166 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:40.595132 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:42.213801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:42.213765 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:43.802427 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:43.802392 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:45.749107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:45.749068 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:47.551622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:47.548788 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:49.807584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:49.807478 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:47:49.816706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:49.816664 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:47:50.198846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:50.198744 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:51.914525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:51.914487 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:53.061352 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:47:53.061307 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:47:53.536841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:53.536808 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:55.219906 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:55.219873 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:56.602301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:56.602269 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:56.884954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:56.884840 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:58.596695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:47:58.596624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:00.319631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:00.317875 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:01.843742 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:01.843703 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:01.999880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:01.999837 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:02.181941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:02.181851 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:02.361215 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:02.360680 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:02.515626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:02.509173 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:02.888837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:02.888755 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:03.612936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:03.612900 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:03.879423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:03.879334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:04.054162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:04.054123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:04.248882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:04.248847 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:04.439640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:04.439590 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:06.103389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:06.103340 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:06.360557 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:06.360466 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:07.263265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:07.263232 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:07.810758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:07.810724 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:09.019624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.019572 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:09.157577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.157535 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd"] Oct 30 13:48:09.158494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158469 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:48:09.158494 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158492 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:48:09.158720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158542 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f78c8b83-98e3-4ef8-8620-22288b1a2c9f" containerName="payload-processing" Oct 30 13:48:09.158720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158550 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78c8b83-98e3-4ef8-8620-22288b1a2c9f" containerName="payload-processing" Oct 30 13:48:09.158720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158577 5693 cpu_manager.go:414] "containerMap: removing container" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:48:09.158720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158585 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:48:09.158720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158631 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:48:09.158720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158639 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:48:09.159035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158832 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cafd9d-d290-48e1-98f6-879e09b69c93" containerName="cleanup" Oct 30 13:48:09.159035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158853 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78c8b83-98e3-4ef8-8620-22288b1a2c9f" containerName="payload-processing" Oct 30 13:48:09.159035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158868 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="795cf0d1-0c49-42bd-b774-03f38280531d" containerName="cleanup" Oct 30 13:48:09.159035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158882 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3938fe33-e533-40a0-a039-e3fe6a52cd1e" containerName="cleanup" Oct 30 13:48:09.159035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.158900 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede01344-c35a-4f03-8850-065c68616c6b" containerName="cleanup" Oct 30 13:48:09.163076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.163055 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.194813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.194781 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-95dcf14f0c60\"" Oct 30 13:48:09.195069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.194804 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"openshift-service-ca.crt\"" Oct 30 13:48:09.195069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.194804 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cad2c6\"" Oct 30 13:48:09.195069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.194846 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"kube-root-ca.crt\"" Oct 30 13:48:09.195334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.194846 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-55rvm\"" Oct 30 13:48:09.195334 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.194967 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"routing-overlay\"" Oct 30 13:48:09.200354 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.200324 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd"] Oct 30 13:48:09.241055 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.241012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.241237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.241090 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.241336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.241313 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.241376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.241348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwg2\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.347682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.346837 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.347682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.346900 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwg2\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.347682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.347075 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.347682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.347136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.348625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.348362 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.361626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.349318 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.364971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.364937 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.385678 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.385641 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwg2\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.477911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.477874 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:09.606193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.606063 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:09.900625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:09.900566 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd"] Oct 30 13:48:09.903191 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:48:09.903160 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ca85d9_c97d_4393_a088_6a08913d4409.slice/crio-7b9b4b8b0dcc4526ef9381d93a3fed14a9f47ac4ac3955aed03c50d0effff543 WatchSource:0}: Error finding container 7b9b4b8b0dcc4526ef9381d93a3fed14a9f47ac4ac3955aed03c50d0effff543: Status 404 returned error can't find the container with id 7b9b4b8b0dcc4526ef9381d93a3fed14a9f47ac4ac3955aed03c50d0effff543 Oct 30 13:48:10.229127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:10.229071 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" event={"ID":"b1ca85d9-c97d-4393-a088-6a08913d4409","Type":"ContainerStarted","Data":"2ab1a3b5fd4065c281d22ef5d8ef2dd541c0945ad2eb7c493e4b4df89cbf29b4"} Oct 30 13:48:10.229548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:10.229172 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:10.229548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:10.229197 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" event={"ID":"b1ca85d9-c97d-4393-a088-6a08913d4409","Type":"ContainerStarted","Data":"7b9b4b8b0dcc4526ef9381d93a3fed14a9f47ac4ac3955aed03c50d0effff543"} Oct 30 13:48:10.285497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:10.285446 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" podStartSLOduration=1.285432286 podStartE2EDuration="1.285432286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:10.284828138 +0000 UTC m=+2666.151446312" watchObservedRunningTime="2026-10-30 13:48:10.285432286 +0000 UTC m=+2666.152050463" Oct 30 13:48:11.341761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:11.341724 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:13.050817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:13.050667 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:14.063524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.063488 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd"] Oct 30 13:48:14.066382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.066358 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:14.180732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.180697 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:14.330893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.330808 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:14.497355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.497326 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:14.653508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.653301 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:14.830387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:14.823187 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:15.038828 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:15.038782 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:15.192685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:15.192647 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:15.846304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:15.843105 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:17.175800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:17.175751 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:18.757623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:18.754163 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:19.064817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.064725 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" podUID="b1ca85d9-c97d-4393-a088-6a08913d4409" containerName="payload-processing" containerID="cri-o://2ab1a3b5fd4065c281d22ef5d8ef2dd541c0945ad2eb7c493e4b4df89cbf29b4" gracePeriod=25 Oct 30 13:48:19.353551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.353516 5693 generic.go:356] "Generic (PLEG): container finished" podID="b1ca85d9-c97d-4393-a088-6a08913d4409" containerID="2ab1a3b5fd4065c281d22ef5d8ef2dd541c0945ad2eb7c493e4b4df89cbf29b4" exitCode=0 Oct 30 13:48:19.353743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.353568 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" event={"ID":"b1ca85d9-c97d-4393-a088-6a08913d4409","Type":"ContainerDied","Data":"2ab1a3b5fd4065c281d22ef5d8ef2dd541c0945ad2eb7c493e4b4df89cbf29b4"} Oct 30 13:48:19.353743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.353662 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" event={"ID":"b1ca85d9-c97d-4393-a088-6a08913d4409","Type":"ContainerDied","Data":"7b9b4b8b0dcc4526ef9381d93a3fed14a9f47ac4ac3955aed03c50d0effff543"} Oct 30 13:48:19.353743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.353680 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9b4b8b0dcc4526ef9381d93a3fed14a9f47ac4ac3955aed03c50d0effff543" Oct 30 13:48:19.360315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.360292 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:19.510296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.510265 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2\") pod \"b1ca85d9-c97d-4393-a088-6a08913d4409\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " Oct 30 13:48:19.510490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.510349 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume\") pod \"b1ca85d9-c97d-4393-a088-6a08913d4409\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " Oct 30 13:48:19.510490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.510427 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials\") pod \"b1ca85d9-c97d-4393-a088-6a08913d4409\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " Oct 30 13:48:19.510697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.510681 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume\") pod \"b1ca85d9-c97d-4393-a088-6a08913d4409\" (UID: \"b1ca85d9-c97d-4393-a088-6a08913d4409\") " Oct 30 13:48:19.510815 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.510790 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume" pod "b1ca85d9-c97d-4393-a088-6a08913d4409" (UID: "b1ca85d9-c97d-4393-a088-6a08913d4409"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:19.511131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.511107 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume" pod "b1ca85d9-c97d-4393-a088-6a08913d4409" (UID: "b1ca85d9-c97d-4393-a088-6a08913d4409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:19.511279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.511266 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:19.511316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.511282 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1ca85d9-c97d-4393-a088-6a08913d4409-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:19.512563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.512540 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2" pod "b1ca85d9-c97d-4393-a088-6a08913d4409" (UID: "b1ca85d9-c97d-4393-a088-6a08913d4409"). InnerVolumeSpecName "kube-api-access-9qwg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:19.512709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.512568 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials" pod "b1ca85d9-c97d-4393-a088-6a08913d4409" (UID: "b1ca85d9-c97d-4393-a088-6a08913d4409"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:19.613298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.613205 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:19.613298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:19.613237 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qwg2\" (UniqueName: \"kubernetes.io/projected/b1ca85d9-c97d-4393-a088-6a08913d4409-kube-api-access-9qwg2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:20.362227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:20.362189 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd" Oct 30 13:48:20.413816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:20.413778 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd"] Oct 30 13:48:20.426514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:20.426474 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-57d7f4bcd"] Oct 30 13:48:20.444217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:20.444179 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:22.078625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:22.078224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:22.256512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:22.256479 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ca85d9-c97d-4393-a088-6a08913d4409" path="/var/lib/kubelet/pods/b1ca85d9-c97d-4393-a088-6a08913d4409/volumes" Oct 30 13:48:22.320337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:22.320303 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:22.478050 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:22.478017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:22.645530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:22.645489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:22.854237 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:22.854154 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.002153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:23.002118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.064635 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:48:23.063710 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:48:23.197712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:23.185151 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.345706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:23.345669 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.942804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:23.942773 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:24.115562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:24.115532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:24.269624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:24.268675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:24.579228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:24.579149 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:24.736134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:24.736102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:26.091227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:26.091185 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:26.909368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:26.909335 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:27.064915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.064879 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:27.226636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.226112 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:27.314441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.314371 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc"] Oct 30 13:48:27.315243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.315224 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b1ca85d9-c97d-4393-a088-6a08913d4409" containerName="payload-processing" Oct 30 13:48:27.315325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.315245 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ca85d9-c97d-4393-a088-6a08913d4409" containerName="payload-processing" Oct 30 13:48:27.315409 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.315397 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ca85d9-c97d-4393-a088-6a08913d4409" containerName="payload-processing" Oct 30 13:48:27.320648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.320624 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.352227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.352043 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a75d02\"" Oct 30 13:48:27.352227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.352114 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"kube-root-ca.crt\"" Oct 30 13:48:27.352227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.352141 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"routing-overlay\"" Oct 30 13:48:27.352513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.352323 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"openshift-service-ca.crt\"" Oct 30 13:48:27.360133 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.360093 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc"] Oct 30 13:48:27.360536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.360514 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"agc-cred-439f75e8497a\"" Oct 30 13:48:27.360741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.360632 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"payload-processing-external-model-e2e--08fbcc8d-dockercfg-646d9\"" Oct 30 13:48:27.388536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.388478 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:27.500675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.498750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.500675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.498804 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsg6\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.500675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.498885 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.500675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.499135 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.560865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.560832 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:27.604201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.604159 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.604761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.604728 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.605068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.605051 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.605205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.605189 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsg6\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.625619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.610771 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.625619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.611961 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.638434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.638401 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.656846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.643325 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsg6\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:27.943967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:27.943880 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:28.016378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:28.016346 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:28.261195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:28.260704 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:28.515348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:28.515301 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:28.575630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:28.572801 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc"] Oct 30 13:48:29.701888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:29.701846 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" event={"ID":"c57e4724-c411-4e13-b8ba-6a63080d39b6","Type":"ContainerStarted","Data":"587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933"} Oct 30 13:48:29.702419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:29.701930 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" event={"ID":"c57e4724-c411-4e13-b8ba-6a63080d39b6","Type":"ContainerStarted","Data":"fea369fa558e9e618ae5ce3b0fed8d61ec1b6aa0335a5949e0600498468b5bfd"} Oct 30 13:48:29.702419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:29.701956 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:29.743110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:29.743046 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" podStartSLOduration=2.743028841 podStartE2EDuration="2.743028841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:29.739935852 +0000 UTC m=+2685.606554025" watchObservedRunningTime="2026-10-30 13:48:29.743028841 +0000 UTC m=+2685.609647018" Oct 30 13:48:29.938533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:29.938496 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.262682 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.262644 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.307810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.307778 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw"] Oct 30 13:48:31.312307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.312285 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.331882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.331835 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5494c9\"" Oct 30 13:48:31.337093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.337063 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw"] Oct 30 13:48:31.394524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.394489 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mzl\" (UniqueName: \"kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl\") pod \"payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.394786 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.394762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.404231 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.404201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.503695 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.503646 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.512803 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.512719 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mzl\" (UniqueName: \"kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl\") pod \"payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.513688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.504943 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.558008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.557975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mzl\" (UniqueName: \"kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl\") pod \"payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.642693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.641314 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:31.825583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.825484 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.970240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.970192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.994974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:31.994939 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw"] Oct 30 13:48:32.383763 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.383732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:32.734091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.734062 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" event={"ID":"bc314fbc-93c7-453b-9dfc-1566797dbfa6","Type":"ContainerStarted","Data":"33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c"} Oct 30 13:48:32.734256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.734110 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" event={"ID":"bc314fbc-93c7-453b-9dfc-1566797dbfa6","Type":"ContainerStarted","Data":"fb11514a2df294e694f732bd922254b97c78b7344e6c51aba8bcf47a10202248"} Oct 30 13:48:32.734303 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.734247 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:32.774022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.773970 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" podStartSLOduration=1.773955594 podStartE2EDuration="1.773955594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:32.771285546 +0000 UTC m=+2688.637903720" watchObservedRunningTime="2026-10-30 13:48:32.773955594 +0000 UTC m=+2688.640573795" Oct 30 13:48:32.861185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.861146 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg"] Oct 30 13:48:32.866193 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.866166 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:32.900622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.900086 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\"" Oct 30 13:48:32.900911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.900885 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\"" Oct 30 13:48:32.901059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.900975 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-f06454846ef2\"" Oct 30 13:48:32.901059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.900986 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\"" Oct 30 13:48:32.901059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.901047 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\"" Oct 30 13:48:32.901059 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.901058 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-10e358d33ffb\"" Oct 30 13:48:32.901307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.901178 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-59xr6\"" Oct 30 13:48:32.911130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.911083 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg"] Oct 30 13:48:32.963224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.961436 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:32.963224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.961506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:32.963224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.961686 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkb5\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:32.963224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:32.961967 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.064080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.063992 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.064080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.064033 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.064286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.064085 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkb5\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.064286 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.064178 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.064918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.064886 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.065047 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.065030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.066692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.066671 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.083052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.083028 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkb5\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.198353 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.198320 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.518117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.518087 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg"] Oct 30 13:48:33.520987 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:48:33.520955 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6042da53_8bfe_484b_84fd_17b27fd80184.slice/crio-27ef794d42b8d4e39ec176c9108f2695c8fe425de12e2d5d2762f0b4051fee0c WatchSource:0}: Error finding container 27ef794d42b8d4e39ec176c9108f2695c8fe425de12e2d5d2762f0b4051fee0c: Status 404 returned error can't find the container with id 27ef794d42b8d4e39ec176c9108f2695c8fe425de12e2d5d2762f0b4051fee0c Oct 30 13:48:33.626813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.626768 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:33.744219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.744179 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" event={"ID":"6042da53-8bfe-484b-84fd-17b27fd80184","Type":"ContainerStarted","Data":"ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167"} Oct 30 13:48:33.744405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.744245 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" event={"ID":"6042da53-8bfe-484b-84fd-17b27fd80184","Type":"ContainerStarted","Data":"27ef794d42b8d4e39ec176c9108f2695c8fe425de12e2d5d2762f0b4051fee0c"} Oct 30 13:48:33.744484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.744409 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:33.787094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:33.786941 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" podStartSLOduration=1.786923493 podStartE2EDuration="1.786923493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:33.785499447 +0000 UTC m=+2689.652117630" watchObservedRunningTime="2026-10-30 13:48:33.786923493 +0000 UTC m=+2689.653541666" Oct 30 13:48:34.073014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:34.072935 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc"] Oct 30 13:48:34.075896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:34.075872 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:34.186697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:34.186661 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:35.407735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:35.407694 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:37.135049 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:37.135014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:38.822510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:38.822477 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:39.073651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.073531 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" podUID="c57e4724-c411-4e13-b8ba-6a63080d39b6" containerName="payload-processing" containerID="cri-o://587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933" gracePeriod=25 Oct 30 13:48:39.352033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.352003 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:39.480835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.480798 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6\") pod \"c57e4724-c411-4e13-b8ba-6a63080d39b6\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " Oct 30 13:48:39.481008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.480916 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume\") pod \"c57e4724-c411-4e13-b8ba-6a63080d39b6\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " Oct 30 13:48:39.481093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.481077 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials\") pod \"c57e4724-c411-4e13-b8ba-6a63080d39b6\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " Oct 30 13:48:39.481184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.481164 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume\") pod \"c57e4724-c411-4e13-b8ba-6a63080d39b6\" (UID: \"c57e4724-c411-4e13-b8ba-6a63080d39b6\") " Oct 30 13:48:39.481347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.481325 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume" pod "c57e4724-c411-4e13-b8ba-6a63080d39b6" (UID: "c57e4724-c411-4e13-b8ba-6a63080d39b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:39.481589 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.481562 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume" pod "c57e4724-c411-4e13-b8ba-6a63080d39b6" (UID: "c57e4724-c411-4e13-b8ba-6a63080d39b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:39.481865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.481847 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:39.481951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.481866 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c57e4724-c411-4e13-b8ba-6a63080d39b6-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:39.483042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.483018 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6" pod "c57e4724-c411-4e13-b8ba-6a63080d39b6" (UID: "c57e4724-c411-4e13-b8ba-6a63080d39b6"). InnerVolumeSpecName "kube-api-access-wgsg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:39.483236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.483215 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials" pod "c57e4724-c411-4e13-b8ba-6a63080d39b6" (UID: "c57e4724-c411-4e13-b8ba-6a63080d39b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:39.583767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.583676 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsg6\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-kube-api-access-wgsg6\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:39.583767 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.583709 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c57e4724-c411-4e13-b8ba-6a63080d39b6-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:39.792699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.792658 5693 generic.go:356] "Generic (PLEG): container finished" podID="c57e4724-c411-4e13-b8ba-6a63080d39b6" containerID="587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933" exitCode=0 Oct 30 13:48:39.792892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.792708 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" event={"ID":"c57e4724-c411-4e13-b8ba-6a63080d39b6","Type":"ContainerDied","Data":"587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933"} Oct 30 13:48:39.792892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.792715 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" Oct 30 13:48:39.792892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.792834 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc" event={"ID":"c57e4724-c411-4e13-b8ba-6a63080d39b6","Type":"ContainerDied","Data":"fea369fa558e9e618ae5ce3b0fed8d61ec1b6aa0335a5949e0600498468b5bfd"} Oct 30 13:48:39.792892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.792861 5693 scope.go:122] "RemoveContainer" containerID="587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933" Oct 30 13:48:39.804374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.804353 5693 scope.go:122] "RemoveContainer" containerID="587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933" Oct 30 13:48:39.804724 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:48:39.804699 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933\": container with ID starting with 587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933 not found: ID does not exist" containerID="587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933" Oct 30 13:48:39.804805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.804731 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933"} err="failed to get container status \"587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933\": rpc error: code = NotFound desc = could not find container \"587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933\": container with ID starting with 587e47e82a58b8ebde474fc185270c98276bb7b8aa4066577f2c165a930fc933 not found: ID does not exist" Oct 30 13:48:39.859314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.859239 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc"] Oct 30 13:48:39.865747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.865720 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-6897xvhpc"] Oct 30 13:48:39.909302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:39.909175 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:40.078099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:40.078060 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:40.258674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:40.258638 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57e4724-c411-4e13-b8ba-6a63080d39b6" path="/var/lib/kubelet/pods/c57e4724-c411-4e13-b8ba-6a63080d39b6/volumes" Oct 30 13:48:40.384861 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:40.384826 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:40.597974 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:40.597878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:40.902676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:40.887867 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:41.074626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:41.067162 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:42.280540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:42.280496 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66"] Oct 30 13:48:42.343170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:42.343133 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:43.747052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:43.747018 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:48:44.171745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:44.171666 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:44.756162 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:44.756125 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:48:44.821383 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:44.821349 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:45.073702 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:45.073583 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:46.116620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:46.113635 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:47.055201 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.055156 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:47.312950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.302659 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2"] Oct 30 13:48:47.314878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.314843 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c57e4724-c411-4e13-b8ba-6a63080d39b6" containerName="payload-processing" Oct 30 13:48:47.315062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.315046 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57e4724-c411-4e13-b8ba-6a63080d39b6" containerName="payload-processing" Oct 30 13:48:47.315363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.315346 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57e4724-c411-4e13-b8ba-6a63080d39b6" containerName="payload-processing" Oct 30 13:48:47.345138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.322390 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" podUID="7893b146-058a-40dc-963c-9d2726ffb0f8" containerName="payload-processing" containerID="cri-o://3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d" gracePeriod=25 Oct 30 13:48:47.368502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.347314 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:47.484911 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.481052 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2"] Oct 30 13:48:47.486401 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.482703 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.503211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.502564 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-51605e\"" Oct 30 13:48:47.581114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.581035 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srg2x\" (UniqueName: \"kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x\") pod \"payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.581307 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.581279 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.585660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.585640 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-51605e-dockercfg-5tkc4\"" Oct 30 13:48:47.683213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.683174 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.683446 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.683421 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srg2x\" (UniqueName: \"kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x\") pod \"payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.684007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.683986 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.704723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.704688 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srg2x\" (UniqueName: \"kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x\") pod \"payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.783986 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.783961 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:48:47.813098 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.813068 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:47.869454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.869412 5693 generic.go:356] "Generic (PLEG): container finished" podID="7893b146-058a-40dc-963c-9d2726ffb0f8" containerID="3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d" exitCode=0 Oct 30 13:48:47.869638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.869457 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" Oct 30 13:48:47.869638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.869488 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" event={"ID":"7893b146-058a-40dc-963c-9d2726ffb0f8","Type":"ContainerDied","Data":"3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d"} Oct 30 13:48:47.869638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.869546 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66" event={"ID":"7893b146-058a-40dc-963c-9d2726ffb0f8","Type":"ContainerDied","Data":"e858c90e7c3195b393085df1b10633316f2204b8da9bc385a2ad3be142df65b8"} Oct 30 13:48:47.869638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.869576 5693 scope.go:122] "RemoveContainer" containerID="3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d" Oct 30 13:48:47.881730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.881703 5693 scope.go:122] "RemoveContainer" containerID="3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d" Oct 30 13:48:47.882053 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:48:47.882031 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d\": container with ID starting with 3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d not found: ID does not exist" containerID="3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d" Oct 30 13:48:47.882150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.882061 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d"} err="failed to get container status \"3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d\": rpc error: code = NotFound desc = could not find container \"3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d\": container with ID starting with 3d34c870e335caa1ef5302d41274887bde6a25f1b22f3a38779c793b1f53ff1d not found: ID does not exist" Oct 30 13:48:47.887095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.886322 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials\") pod \"7893b146-058a-40dc-963c-9d2726ffb0f8\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " Oct 30 13:48:47.887227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.887151 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj\") pod \"7893b146-058a-40dc-963c-9d2726ffb0f8\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " Oct 30 13:48:47.890360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.887499 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume\") pod \"7893b146-058a-40dc-963c-9d2726ffb0f8\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " Oct 30 13:48:47.890360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.887558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume\") pod \"7893b146-058a-40dc-963c-9d2726ffb0f8\" (UID: \"7893b146-058a-40dc-963c-9d2726ffb0f8\") " Oct 30 13:48:47.890360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.889534 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials" pod "7893b146-058a-40dc-963c-9d2726ffb0f8" (UID: "7893b146-058a-40dc-963c-9d2726ffb0f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:47.890360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.889541 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume" pod "7893b146-058a-40dc-963c-9d2726ffb0f8" (UID: "7893b146-058a-40dc-963c-9d2726ffb0f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:47.890360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.889578 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume" pod "7893b146-058a-40dc-963c-9d2726ffb0f8" (UID: "7893b146-058a-40dc-963c-9d2726ffb0f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:47.891167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.890999 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj" pod "7893b146-058a-40dc-963c-9d2726ffb0f8" (UID: "7893b146-058a-40dc-963c-9d2726ffb0f8"). InnerVolumeSpecName "kube-api-access-jjqlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:47.989762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.989728 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:47.989762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.989759 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7893b146-058a-40dc-963c-9d2726ffb0f8-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:47.989970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.989772 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:47.989970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:47.989785 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjqlj\" (UniqueName: \"kubernetes.io/projected/7893b146-058a-40dc-963c-9d2726ffb0f8-kube-api-access-jjqlj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:48:48.162007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.161976 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2"] Oct 30 13:48:48.164302 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:48:48.164274 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb842c40d_bab5_47fe_b667_f718da5cfa3b.slice/crio-377dbb4ec417116f9b1d32284c07121e887694b81ae9dfb608615abf341c4ab1 WatchSource:0}: Error finding container 377dbb4ec417116f9b1d32284c07121e887694b81ae9dfb608615abf341c4ab1: Status 404 returned error can't find the container with id 377dbb4ec417116f9b1d32284c07121e887694b81ae9dfb608615abf341c4ab1 Oct 30 13:48:48.178577 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.178548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:48.226842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.226428 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66"] Oct 30 13:48:48.241711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.241483 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-59642mb66"] Oct 30 13:48:48.260149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.260116 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7893b146-058a-40dc-963c-9d2726ffb0f8" path="/var/lib/kubelet/pods/7893b146-058a-40dc-963c-9d2726ffb0f8/volumes" Oct 30 13:48:48.880431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.880337 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" event={"ID":"b842c40d-bab5-47fe-b667-f718da5cfa3b","Type":"ContainerStarted","Data":"5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29"} Oct 30 13:48:48.880431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.880408 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:48:48.880431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.880430 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" event={"ID":"b842c40d-bab5-47fe-b667-f718da5cfa3b","Type":"ContainerStarted","Data":"377dbb4ec417116f9b1d32284c07121e887694b81ae9dfb608615abf341c4ab1"} Oct 30 13:48:48.935644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:48.935574 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" podStartSLOduration=1.935556113 podStartE2EDuration="1.935556113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:48.933188874 +0000 UTC m=+2704.799807060" watchObservedRunningTime="2026-10-30 13:48:48.935556113 +0000 UTC m=+2704.802174285" Oct 30 13:48:49.950748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:49.950272 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:51.758495 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:51.758454 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:53.060238 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:48:53.060203 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:48:53.487636 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:53.484867 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:55.228434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:55.228354 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.689625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:56.689229 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:57.096065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:57.096030 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.864782 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:58.863021 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:59.895584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:48:59.895552 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:49:00.508619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:00.505105 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:01.031845 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:01.031811 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:02.420236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.420171 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:02.452629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.452568 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s"] Oct 30 13:49:02.454136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.454077 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7893b146-058a-40dc-963c-9d2726ffb0f8" containerName="payload-processing" Oct 30 13:49:02.454824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.454801 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7893b146-058a-40dc-963c-9d2726ffb0f8" containerName="payload-processing" Oct 30 13:49:02.455300 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.455283 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7893b146-058a-40dc-963c-9d2726ffb0f8" containerName="payload-processing" Oct 30 13:49:02.463022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.462993 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.493417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.493381 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s"] Oct 30 13:49:02.495996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.495971 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-5607248d5d3d\"" Oct 30 13:49:02.508070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.508039 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2cecb\"" Oct 30 13:49:02.508070 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.508061 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"kube-root-ca.crt\"" Oct 30 13:49:02.508279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.508061 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-e2e--cb40fe81-dockercfg-z6lx5\"" Oct 30 13:49:02.508279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.508127 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"routing-overlay\"" Oct 30 13:49:02.508420 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.508403 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"openshift-service-ca.crt\"" Oct 30 13:49:02.603406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.603371 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.603633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.603430 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28cn\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.603633 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.603520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.603766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.603742 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.705211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.705122 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.705381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.705319 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.705422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.705403 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.705463 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.705442 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28cn\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.706078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.706048 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.706207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.706091 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.707938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.707912 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.727218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.727185 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28cn\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:02.813356 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:02.813314 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:03.134663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:03.134633 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s"] Oct 30 13:49:03.136591 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:49:03.136562 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46d4fe6_ba91_423a_a74e_f464c8a71adc.slice/crio-61007fc2d4a7b96fc37f6f41ee23e436fd37ec3125dafccf7a4ff9c5a1b4be53 WatchSource:0}: Error finding container 61007fc2d4a7b96fc37f6f41ee23e436fd37ec3125dafccf7a4ff9c5a1b4be53: Status 404 returned error can't find the container with id 61007fc2d4a7b96fc37f6f41ee23e436fd37ec3125dafccf7a4ff9c5a1b4be53 Oct 30 13:49:04.007818 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:04.007772 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" event={"ID":"e46d4fe6-ba91-423a-a74e-f464c8a71adc","Type":"ContainerStarted","Data":"4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110"} Oct 30 13:49:04.008221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:04.007854 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" event={"ID":"e46d4fe6-ba91-423a-a74e-f464c8a71adc","Type":"ContainerStarted","Data":"61007fc2d4a7b96fc37f6f41ee23e436fd37ec3125dafccf7a4ff9c5a1b4be53"} Oct 30 13:49:04.008221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:04.007891 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:04.055205 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:04.055147 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" podStartSLOduration=2.05513007 podStartE2EDuration="2.05513007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:04.052697081 +0000 UTC m=+2719.919315254" watchObservedRunningTime="2026-10-30 13:49:04.05513007 +0000 UTC m=+2719.921748244" Oct 30 13:49:04.268351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:04.267834 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:06.026297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:06.026259 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:07.669073 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:07.669009 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:09.337249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:09.337215 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:11.083183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:11.083135 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:12.629227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:12.629182 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.284272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:14.284215 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.425026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:14.424990 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.586510 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:14.586417 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:15.018869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:15.018839 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:17.166243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:17.166210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:17.958698 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:17.958656 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:19.713432 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:19.713386 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:21.303797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:21.303749 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:21.947470 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:21.947424 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:21.967314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:21.967286 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s"] Oct 30 13:49:22.086752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:22.086718 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:23.056669 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:49:23.056633 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:49:23.206854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:23.206791 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:24.846028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:24.845998 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:26.525616 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:26.525566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:26.967799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:26.967712 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" podUID="e46d4fe6-ba91-423a-a74e-f464c8a71adc" containerName="payload-processing" containerID="cri-o://4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110" gracePeriod=25 Oct 30 13:49:27.271304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.271275 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:27.293920 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.293891 5693 generic.go:356] "Generic (PLEG): container finished" podID="e46d4fe6-ba91-423a-a74e-f464c8a71adc" containerID="4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110" exitCode=0 Oct 30 13:49:27.294069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.293940 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" Oct 30 13:49:27.294069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.293944 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" event={"ID":"e46d4fe6-ba91-423a-a74e-f464c8a71adc","Type":"ContainerDied","Data":"4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110"} Oct 30 13:49:27.294151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.294082 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s" event={"ID":"e46d4fe6-ba91-423a-a74e-f464c8a71adc","Type":"ContainerDied","Data":"61007fc2d4a7b96fc37f6f41ee23e436fd37ec3125dafccf7a4ff9c5a1b4be53"} Oct 30 13:49:27.294151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.294109 5693 scope.go:122] "RemoveContainer" containerID="4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110" Oct 30 13:49:27.304405 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.304386 5693 scope.go:122] "RemoveContainer" containerID="4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110" Oct 30 13:49:27.304682 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:49:27.304656 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110\": container with ID starting with 4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110 not found: ID does not exist" containerID="4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110" Oct 30 13:49:27.304739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.304681 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110"} err="failed to get container status \"4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110\": rpc error: code = NotFound desc = could not find container \"4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110\": container with ID starting with 4a4bb8333e25244fa96b7251b21b1bf32a675a0c713d006259b9170fcd0d2110 not found: ID does not exist" Oct 30 13:49:27.433493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.433458 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume\") pod \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " Oct 30 13:49:27.433712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.433621 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials\") pod \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " Oct 30 13:49:27.433712 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.433694 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume\") pod \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " Oct 30 13:49:27.433813 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.433800 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn\") pod \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\" (UID: \"e46d4fe6-ba91-423a-a74e-f464c8a71adc\") " Oct 30 13:49:27.434010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.433980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume" pod "e46d4fe6-ba91-423a-a74e-f464c8a71adc" (UID: "e46d4fe6-ba91-423a-a74e-f464c8a71adc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:27.434065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.434022 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume" pod "e46d4fe6-ba91-423a-a74e-f464c8a71adc" (UID: "e46d4fe6-ba91-423a-a74e-f464c8a71adc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:27.434370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.434338 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:27.434370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.434361 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46d4fe6-ba91-423a-a74e-f464c8a71adc-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:27.435882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.435860 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials" pod "e46d4fe6-ba91-423a-a74e-f464c8a71adc" (UID: "e46d4fe6-ba91-423a-a74e-f464c8a71adc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:27.436016 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.435995 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn" pod "e46d4fe6-ba91-423a-a74e-f464c8a71adc" (UID: "e46d4fe6-ba91-423a-a74e-f464c8a71adc"). InnerVolumeSpecName "kube-api-access-m28cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:27.536292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.536216 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:27.536292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.536243 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m28cn\" (UniqueName: \"kubernetes.io/projected/e46d4fe6-ba91-423a-a74e-f464c8a71adc-kube-api-access-m28cn\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:27.640445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.640415 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s"] Oct 30 13:49:27.644032 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:27.644008 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-cdc4wd47s"] Oct 30 13:49:28.197253 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:28.197216 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:28.268048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:28.268009 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46d4fe6-ba91-423a-a74e-f464c8a71adc" path="/var/lib/kubelet/pods/e46d4fe6-ba91-423a-a74e-f464c8a71adc/volumes" Oct 30 13:49:29.763169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:29.763131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:31.586621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.582415 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg"] Oct 30 13:49:31.586621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.584407 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e46d4fe6-ba91-423a-a74e-f464c8a71adc" containerName="payload-processing" Oct 30 13:49:31.586621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.584430 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46d4fe6-ba91-423a-a74e-f464c8a71adc" containerName="payload-processing" Oct 30 13:49:31.606051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.606019 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46d4fe6-ba91-423a-a74e-f464c8a71adc" containerName="payload-processing" Oct 30 13:49:31.614647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.614615 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg"] Oct 30 13:49:31.615052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.614975 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.631503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.631473 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-51605e-serving-cert\"" Oct 30 13:49:31.752364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.752329 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.752625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.752589 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcdx\" (UniqueName: \"kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.752686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.752641 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.752686 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.752677 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.766145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.766114 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:31.864673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.857785 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.864673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.858078 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.864673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.858283 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcdx\" (UniqueName: \"kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.864673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.858323 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.897555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.897514 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.899242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.899207 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.899389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.899337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.904755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.904726 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:31.929107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.929059 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcdx\" (UniqueName: \"kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx\") pod \"maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:31.962835 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:31.962807 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:32.272408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:32.272382 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg"] Oct 30 13:49:32.274668 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:49:32.274643 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f86acb_0da5_4154_a4f8_c8dfbbb0ea18.slice/crio-bab988fc789ab0583c2306e92137ff02f9d41fde79f4e83f08d7a929e4290fb5 WatchSource:0}: Error finding container bab988fc789ab0583c2306e92137ff02f9d41fde79f4e83f08d7a929e4290fb5: Status 404 returned error can't find the container with id bab988fc789ab0583c2306e92137ff02f9d41fde79f4e83f08d7a929e4290fb5 Oct 30 13:49:32.275963 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:32.275942 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:49:32.372110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:32.372071 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" event={"ID":"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18","Type":"ContainerStarted","Data":"bab988fc789ab0583c2306e92137ff02f9d41fde79f4e83f08d7a929e4290fb5"} Oct 30 13:49:33.382644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:33.382586 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" event={"ID":"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18","Type":"ContainerStarted","Data":"1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9"} Oct 30 13:49:33.383051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:33.382694 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:33.428872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:33.428807 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" podStartSLOduration=2.0469972 podStartE2EDuration="2.428789036s" totalImagesPullingTime="381.791836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:33.42736471 +0000 UTC m=+2749.293982895" watchObservedRunningTime="2026-10-30 13:49:33.428789036 +0000 UTC m=+2749.295407209" Oct 30 13:49:33.823661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:33.823564 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:34.475085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:34.475052 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:35.125519 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:35.125480 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:35.298170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:35.298116 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:35.778186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:35.778149 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:37.553056 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:37.552210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:38.009127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.009094 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79"] Oct 30 13:49:38.014397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.014376 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.041784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.041763 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"routing-overlay\"" Oct 30 13:49:38.051978 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.051957 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"openshift-service-ca.crt\"" Oct 30 13:49:38.054690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.054665 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-8d03b77200d7\"" Oct 30 13:49:38.054834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.054695 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-f7c959f42792\"" Oct 30 13:49:38.054834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.054774 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-e2e--2547b850-dockercfg-zlgbr\"" Oct 30 13:49:38.056512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.056490 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79"] Oct 30 13:49:38.118145 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.118113 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.118331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.118271 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.118480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.118452 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.118579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.118560 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v7q4\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.119338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.119310 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"kube-root-ca.crt\"" Oct 30 13:49:38.119338 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.119310 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3eb98\"" Oct 30 13:49:38.220476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.220444 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v7q4\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.220664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.220594 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.220735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.220721 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.220793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.220782 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.221473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.221451 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.221521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.221468 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.223315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.223293 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.240669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.240645 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v7q4\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.329622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.329526 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:38.638455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:38.638420 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79"] Oct 30 13:49:38.639877 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:49:38.639842 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2647c373_ae8d_4d49_894a_1dffd985bca2.slice/crio-7b40c159cc54ec94be2f4b0612c3740af7bbb963d41a62587f8b778be7c845b0 WatchSource:0}: Error finding container 7b40c159cc54ec94be2f4b0612c3740af7bbb963d41a62587f8b778be7c845b0: Status 404 returned error can't find the container with id 7b40c159cc54ec94be2f4b0612c3740af7bbb963d41a62587f8b778be7c845b0 Oct 30 13:49:39.299542 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:39.299506 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:39.398069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:39.398039 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:49:39.606535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:39.606458 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" event={"ID":"2647c373-ae8d-4d49-894a-1dffd985bca2","Type":"ContainerStarted","Data":"7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597"} Oct 30 13:49:39.606535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:39.606512 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" event={"ID":"2647c373-ae8d-4d49-894a-1dffd985bca2","Type":"ContainerStarted","Data":"7b40c159cc54ec94be2f4b0612c3740af7bbb963d41a62587f8b778be7c845b0"} Oct 30 13:49:39.606850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:39.606539 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:39.645134 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:39.645083 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" podStartSLOduration=2.645062938 podStartE2EDuration="2.645062938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:39.64399997 +0000 UTC m=+2755.510618144" watchObservedRunningTime="2026-10-30 13:49:39.645062938 +0000 UTC m=+2755.511681185" Oct 30 13:49:40.996228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:40.996190 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:41.167268 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:41.167234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:41.322472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:41.322394 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:41.532472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:41.532438 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:41.920466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:41.920429 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:42.087627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:42.087163 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:42.234168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:42.234131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:42.806661 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:42.806624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:43.122181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:43.122097 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:43.258113 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:43.258073 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79"] Oct 30 13:49:43.261772 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:43.261746 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:43.334535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:43.334502 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:44.830345 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:44.830310 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:46.548567 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:46.548535 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:48.256390 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.256357 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:48.259155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.259130 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" podUID="2647c373-ae8d-4d49-894a-1dffd985bca2" containerName="payload-processing" containerID="cri-o://7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597" gracePeriod=25 Oct 30 13:49:48.409362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.409326 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:48.683966 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.683877 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:48.781824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.781736 5693 generic.go:356] "Generic (PLEG): container finished" podID="2647c373-ae8d-4d49-894a-1dffd985bca2" containerID="7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597" exitCode=0 Oct 30 13:49:48.782248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.782134 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" Oct 30 13:49:48.783468 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.782157 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" event={"ID":"2647c373-ae8d-4d49-894a-1dffd985bca2","Type":"ContainerDied","Data":"7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597"} Oct 30 13:49:48.786919 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.786878 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79" event={"ID":"2647c373-ae8d-4d49-894a-1dffd985bca2","Type":"ContainerDied","Data":"7b40c159cc54ec94be2f4b0612c3740af7bbb963d41a62587f8b778be7c845b0"} Oct 30 13:49:48.828703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.828670 5693 scope.go:122] "RemoveContainer" containerID="7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597" Oct 30 13:49:48.873099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.873073 5693 scope.go:122] "RemoveContainer" containerID="7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597" Oct 30 13:49:48.873463 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:49:48.873437 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597\": container with ID starting with 7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597 not found: ID does not exist" containerID="7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597" Oct 30 13:49:48.873617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.873474 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597"} err="failed to get container status \"7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597\": rpc error: code = NotFound desc = could not find container \"7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597\": container with ID starting with 7cd245404ac6e4f1b458730c4081919b13e89ce3146355b8d0f952320a31a597 not found: ID does not exist" Oct 30 13:49:48.879397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.879373 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials\") pod \"2647c373-ae8d-4d49-894a-1dffd985bca2\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " Oct 30 13:49:48.879521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.879478 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume\") pod \"2647c373-ae8d-4d49-894a-1dffd985bca2\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " Oct 30 13:49:48.879593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.879556 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4\") pod \"2647c373-ae8d-4d49-894a-1dffd985bca2\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " Oct 30 13:49:48.879784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.879769 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume\") pod \"2647c373-ae8d-4d49-894a-1dffd985bca2\" (UID: \"2647c373-ae8d-4d49-894a-1dffd985bca2\") " Oct 30 13:49:48.879870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.879837 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume" pod "2647c373-ae8d-4d49-894a-1dffd985bca2" (UID: "2647c373-ae8d-4d49-894a-1dffd985bca2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:48.880210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.880183 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume" pod "2647c373-ae8d-4d49-894a-1dffd985bca2" (UID: "2647c373-ae8d-4d49-894a-1dffd985bca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:48.880477 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.880456 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:48.880552 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.880481 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2647c373-ae8d-4d49-894a-1dffd985bca2-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:48.894796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.894769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials" pod "2647c373-ae8d-4d49-894a-1dffd985bca2" (UID: "2647c373-ae8d-4d49-894a-1dffd985bca2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:48.896186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.896157 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4" pod "2647c373-ae8d-4d49-894a-1dffd985bca2" (UID: "2647c373-ae8d-4d49-894a-1dffd985bca2"). InnerVolumeSpecName "kube-api-access-2v7q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:48.981957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.981917 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:48.981957 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:48.981952 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v7q4\" (UniqueName: \"kubernetes.io/projected/2647c373-ae8d-4d49-894a-1dffd985bca2-kube-api-access-2v7q4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:49:49.132277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:49.132215 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79"] Oct 30 13:49:49.135991 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:49.135966 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-7c96vxt79"] Oct 30 13:49:49.675478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:49.675132 5693 scope.go:122] "RemoveContainer" containerID="6c3a29865e9cfe6212ca57bacf0b8189ac3629184eb16e83f26a31a13e55022a" Oct 30 13:49:50.208363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:50.208335 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:50.257615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:50.257571 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2647c373-ae8d-4d49-894a-1dffd985bca2" path="/var/lib/kubelet/pods/2647c373-ae8d-4d49-894a-1dffd985bca2/volumes" Oct 30 13:49:50.358762 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:50.358727 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:51.199617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.199552 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:51.419155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.419125 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v"] Oct 30 13:49:51.420079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.420060 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2647c373-ae8d-4d49-894a-1dffd985bca2" containerName="payload-processing" Oct 30 13:49:51.420079 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.420079 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2647c373-ae8d-4d49-894a-1dffd985bca2" containerName="payload-processing" Oct 30 13:49:51.420256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.420244 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2647c373-ae8d-4d49-894a-1dffd985bca2" containerName="payload-processing" Oct 30 13:49:51.425848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.425830 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.442108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.442079 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v"] Oct 30 13:49:51.449301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.449274 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"openshift-service-ca.crt\"" Oct 30 13:49:51.449439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.449359 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2cecb\"" Oct 30 13:49:51.449439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.449398 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-f5ed3d5b66e9\"" Oct 30 13:49:51.455092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.454989 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"kube-root-ca.crt\"" Oct 30 13:49:51.455092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.455012 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"routing-overlay\"" Oct 30 13:49:51.455092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.455018 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-e2e--cb40fe81-dockercfg-2p8gp\"" Oct 30 13:49:51.466124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.466097 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847tr\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.466262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.466246 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.466328 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.466300 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.466441 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.466414 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.574677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.574630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847tr\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.575591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.575560 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.576688 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.576668 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.577266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.577248 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.578261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.577913 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.589188 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.576578 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.589335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.589121 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.602521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.602488 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847tr\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:51.741161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:51.741127 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:52.024312 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:52.024191 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:52.088279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:52.088240 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v"] Oct 30 13:49:52.824276 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:52.824236 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" event={"ID":"3e7e6974-db72-4989-a02f-c7887afc9fa7","Type":"ContainerStarted","Data":"2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b"} Oct 30 13:49:52.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:52.824298 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" event={"ID":"3e7e6974-db72-4989-a02f-c7887afc9fa7","Type":"ContainerStarted","Data":"a154b1a83f5f5c6eb8b4821b938459a77a92d8ca0c26d8a31c75f0f0ab7244e5"} Oct 30 13:49:52.824715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:52.824326 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:49:52.865908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:52.865850 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" podStartSLOduration=1.865835793 podStartE2EDuration="1.865835793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:52.861728223 +0000 UTC m=+2768.728346375" watchObservedRunningTime="2026-10-30 13:49:52.865835793 +0000 UTC m=+2768.732453965" Oct 30 13:49:53.072927 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:49:53.072886 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:49:53.756259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:53.755559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:55.464259 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:55.464222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:56.788490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:56.788449 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:57.133711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:57.133618 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:58.753236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:49:58.753186 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:00.350632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:00.347624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:01.946788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:01.942273 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:03.510615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:03.510565 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:03.837107 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:03.837019 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:50:05.245187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:05.245155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:06.376703 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:06.376670 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:06.517886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:06.517844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:06.988081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:06.988040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:07.290052 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:07.289969 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:07.423172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:07.423135 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:09.003229 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:09.003195 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:09.899204 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:09.899159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.032569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.032537 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8"] Oct 30 13:50:10.043880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.043853 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7385aabb-fwrv8"] Oct 30 13:50:10.257027 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.256992 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24980b11-8a6c-407b-895c-696426008772" path="/var/lib/kubelet/pods/24980b11-8a6c-407b-895c-696426008772/volumes" Oct 30 13:50:10.471491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.471453 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.739524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.739468 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.827456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.827415 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs"] Oct 30 13:50:10.841119 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.841086 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.863694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.863578 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs"] Oct 30 13:50:10.881091 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.881057 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-996b8b-openshift-default-dockercfg-th9r6\"" Oct 30 13:50:10.929167 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.929129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.970085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.970035 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.970410 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.970379 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.978630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.970573 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.978936 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.978900 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsvd6\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.979310 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.979281 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.979497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.979478 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.979755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.979737 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.987883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.987850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:10.998884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:10.998796 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106391 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106473 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106617 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106714 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106787 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106840 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsvd6\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.106962 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.107048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.107004 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.108164 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.107502 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.116036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.116001 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.122709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.118637 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.122931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.121777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.123045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.121973 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.132707 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.129340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.149351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.149315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.166938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.166159 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.166938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.166812 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsvd6\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6\") pod \"e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.169210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.168992 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:11.319360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.313792 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:11.616668 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.584623 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs"] Oct 30 13:50:11.636583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.636538 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:50:11.636774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.636673 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:50:11.636774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:11.636725 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:50:12.162185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:12.160914 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" event={"ID":"1836005b-46a2-45c3-b269-75bfbf3cabfb","Type":"ContainerStarted","Data":"15281020e36f866306a87bac0555bc0effee7dceeb9c4d83b7b55de42fc402d6"} Oct 30 13:50:12.162185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:12.160983 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" event={"ID":"1836005b-46a2-45c3-b269-75bfbf3cabfb","Type":"ContainerStarted","Data":"d1472d705d8310c696f2ca97b5ed6db708236c67074e1a5bab62c53b7bbedee8"} Oct 30 13:50:12.174842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:12.174392 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:12.174842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:12.174509 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:50:12.174842 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:12.174549 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:50:12.229138 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:12.229084 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podStartSLOduration=2.229068155 podStartE2EDuration="2.229068155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:12.214080843 +0000 UTC m=+2788.080699016" watchObservedRunningTime="2026-10-30 13:50:12.229068155 +0000 UTC m=+2788.095686356" Oct 30 13:50:13.049207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:13.034348 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:13.175349 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:13.175319 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:14.177745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:14.177710 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:14.178647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:14.178592 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:50:14.278141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:14.278106 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:14.438392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:14.438306 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:14.815849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:14.815808 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:15.033779 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:15.033746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:15.236298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:15.236263 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:15.391650 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:15.390681 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:15.699680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:15.699567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:16.423455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:16.423392 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:16.663018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:16.649852 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:17.339672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:17.332763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:19.196624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:19.185824 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:19.359400 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:19.359362 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:20.962200 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:20.962167 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:21.111793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:21.111754 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:22.701422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:22.701378 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:23.057432 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:50:23.057396 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:50:24.320548 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:24.320514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:26.335745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:26.335666 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:27.231593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:27.231558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:27.389499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:27.389462 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:27.656980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:27.656897 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:28.093631 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:28.093559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:28.284619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:28.274069 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:29.752630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:29.748539 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:31.574788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:31.574743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:32.340051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:32.340016 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:32.503105 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:32.503067 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:33.355714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:33.355620 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:33.971867 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:33.971826 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:34.054447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.054406 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22"] Oct 30 13:50:34.059854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.059828 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.077434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.077407 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22"] Oct 30 13:50:34.085009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.084983 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-88352d\"" Oct 30 13:50:34.098719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.098195 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.098719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.098301 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pb4\" (UniqueName: \"kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4\") pod \"payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.108502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.108471 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm"] Oct 30 13:50:34.113452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.113429 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.124363 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.124331 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:34.135249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.135211 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm"] Oct 30 13:50:34.166620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.164117 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-88352d-dockercfg-srfjn\"" Oct 30 13:50:34.308617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.303177 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.308617 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.303290 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pb4\" (UniqueName: \"kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4\") pod \"payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.373641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.369456 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.373641 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.370623 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pb4\" (UniqueName: \"kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4\") pod \"payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.381404 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.380144 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:34.439116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.438737 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.439116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.438908 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2n2f\" (UniqueName: \"kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f\") pod \"payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.540656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.540626 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.541364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.540859 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2n2f\" (UniqueName: \"kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f\") pod \"payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.541796 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.541777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.559942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.559884 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2n2f\" (UniqueName: \"kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f\") pod \"payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.738716 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.738680 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:34.745865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:34.745837 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22"] Oct 30 13:50:34.752802 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:50:34.752769 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa12f47_cfae_4e2c_afba_686ea76e6b59.slice/crio-8e63a0c916446bdfb36ecdcd64ae579f47e6194953937a71ca9848047976f954 WatchSource:0}: Error finding container 8e63a0c916446bdfb36ecdcd64ae579f47e6194953937a71ca9848047976f954: Status 404 returned error can't find the container with id 8e63a0c916446bdfb36ecdcd64ae579f47e6194953937a71ca9848047976f954 Oct 30 13:50:35.049505 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.049469 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm"] Oct 30 13:50:35.051093 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:50:35.051067 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda349a68f_51c7_4efa_bee6_6d1517db24dd.slice/crio-cedb913106d1cb3f91b38ea3abef1fa7a910ed85ee7d4f3a407234d10e198069 WatchSource:0}: Error finding container cedb913106d1cb3f91b38ea3abef1fa7a910ed85ee7d4f3a407234d10e198069: Status 404 returned error can't find the container with id cedb913106d1cb3f91b38ea3abef1fa7a910ed85ee7d4f3a407234d10e198069 Oct 30 13:50:35.504455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.504418 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:35.548575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.548274 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" event={"ID":"daa12f47-cfae-4e2c-afba-686ea76e6b59","Type":"ContainerStarted","Data":"4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c"} Oct 30 13:50:35.548575 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.548349 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" event={"ID":"daa12f47-cfae-4e2c-afba-686ea76e6b59","Type":"ContainerStarted","Data":"8e63a0c916446bdfb36ecdcd64ae579f47e6194953937a71ca9848047976f954"} Oct 30 13:50:35.548901 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.548784 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:35.550823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.550792 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" event={"ID":"a349a68f-51c7-4efa-bee6-6d1517db24dd","Type":"ContainerStarted","Data":"97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57"} Oct 30 13:50:35.551014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.550845 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" event={"ID":"a349a68f-51c7-4efa-bee6-6d1517db24dd","Type":"ContainerStarted","Data":"cedb913106d1cb3f91b38ea3abef1fa7a910ed85ee7d4f3a407234d10e198069"} Oct 30 13:50:35.551606 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.551573 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:35.588463 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.588394 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" podStartSLOduration=1.588375208 podStartE2EDuration="1.588375208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:35.588082181 +0000 UTC m=+2811.454700359" watchObservedRunningTime="2026-10-30 13:50:35.588375208 +0000 UTC m=+2811.454993381" Oct 30 13:50:35.626182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:35.626082 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" podStartSLOduration=1.6260679599999999 podStartE2EDuration="1.62606796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:35.624229489 +0000 UTC m=+2811.490847662" watchObservedRunningTime="2026-10-30 13:50:35.62606796 +0000 UTC m=+2811.492686132" Oct 30 13:50:36.979036 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:36.979000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:37.130776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:37.130728 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:37.563416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:37.563385 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:38.394914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.394881 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl"] Oct 30 13:50:38.399959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.399937 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.419370 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.419336 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl"] Oct 30 13:50:38.427656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.426268 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"routing-overlay\"" Oct 30 13:50:38.427656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.426288 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"openshift-service-ca.crt\"" Oct 30 13:50:38.427656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.426310 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"payload-processing-external-model-e2e--8b865544-dockercfg-rbmwm\"" Oct 30 13:50:38.427656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.426331 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-98f1cc\"" Oct 30 13:50:38.427656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.426340 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"agc-cred-e539133cb1f7\"" Oct 30 13:50:38.427656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.426515 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"kube-root-ca.crt\"" Oct 30 13:50:38.570199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.570156 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.570386 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.570366 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.570502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.570488 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.570553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.570523 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwmr\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.671706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.671618 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.671706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.671670 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwmr\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.671944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.671764 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.671944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.671900 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.672509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.672476 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.672727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.672707 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.674280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.674258 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.691160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.691128 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwmr\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:38.716369 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:38.716337 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:39.022648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:39.022614 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl"] Oct 30 13:50:39.024705 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:50:39.024676 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf898d20b_efe1_4dbb_88a4_d14fc45cb9eb.slice/crio-439387e7052282b108e058cdb76f24d4c9e02584aaae8e3551d1ef8b5876a0cf WatchSource:0}: Error finding container 439387e7052282b108e058cdb76f24d4c9e02584aaae8e3551d1ef8b5876a0cf: Status 404 returned error can't find the container with id 439387e7052282b108e058cdb76f24d4c9e02584aaae8e3551d1ef8b5876a0cf Oct 30 13:50:39.404136 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:39.404021 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:39.586652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:39.586616 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" event={"ID":"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb","Type":"ContainerStarted","Data":"638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f"} Oct 30 13:50:39.586808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:39.586670 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" event={"ID":"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb","Type":"ContainerStarted","Data":"439387e7052282b108e058cdb76f24d4c9e02584aaae8e3551d1ef8b5876a0cf"} Oct 30 13:50:39.586808 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:39.586709 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:39.627866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:39.627808 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" podStartSLOduration=1.6277945759999999 podStartE2EDuration="1.627794576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:39.624142483 +0000 UTC m=+2815.490760656" watchObservedRunningTime="2026-10-30 13:50:39.627794576 +0000 UTC m=+2815.494412758" Oct 30 13:50:40.964247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:40.964217 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:42.620261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:42.620209 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:44.219017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:44.218979 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:45.813623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:45.809271 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:46.561544 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:46.561513 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:50:47.475290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:47.475216 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:47.591878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:47.591841 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:50:49.549582 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:49.549544 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:49.898769 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:49.898691 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg"] Oct 30 13:50:50.598720 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:50.598686 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:50:51.242029 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:51.241995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:52.927699 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:52.912512 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:53.056654 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:50:53.056616 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:50:54.617591 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:54.617554 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:54.899165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:54.899077 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" podUID="6042da53-8bfe-484b-84fd-17b27fd80184" containerName="payload-processing" containerID="cri-o://ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167" gracePeriod=25 Oct 30 13:50:55.210246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.210217 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:50:55.243512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.243477 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5\") pod \"6042da53-8bfe-484b-84fd-17b27fd80184\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " Oct 30 13:50:55.243748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.243713 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume\") pod \"6042da53-8bfe-484b-84fd-17b27fd80184\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " Oct 30 13:50:55.243834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.243792 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials\") pod \"6042da53-8bfe-484b-84fd-17b27fd80184\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " Oct 30 13:50:55.243894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.243854 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume\") pod \"6042da53-8bfe-484b-84fd-17b27fd80184\" (UID: \"6042da53-8bfe-484b-84fd-17b27fd80184\") " Oct 30 13:50:55.244227 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.244095 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume" pod "6042da53-8bfe-484b-84fd-17b27fd80184" (UID: "6042da53-8bfe-484b-84fd-17b27fd80184"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:50:55.244323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.244303 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume" pod "6042da53-8bfe-484b-84fd-17b27fd80184" (UID: "6042da53-8bfe-484b-84fd-17b27fd80184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:50:55.244799 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.244771 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:50:55.244889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.244800 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6042da53-8bfe-484b-84fd-17b27fd80184-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:50:55.245891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.245862 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5" pod "6042da53-8bfe-484b-84fd-17b27fd80184" (UID: "6042da53-8bfe-484b-84fd-17b27fd80184"). InnerVolumeSpecName "kube-api-access-fbkb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:50:55.246021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.246004 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials" pod "6042da53-8bfe-484b-84fd-17b27fd80184" (UID: "6042da53-8bfe-484b-84fd-17b27fd80184"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:50:55.346672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.346592 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkb5\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-kube-api-access-fbkb5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:50:55.346864 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.346742 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6042da53-8bfe-484b-84fd-17b27fd80184-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:50:55.800728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.800696 5693 generic.go:356] "Generic (PLEG): container finished" podID="6042da53-8bfe-484b-84fd-17b27fd80184" containerID="ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167" exitCode=0 Oct 30 13:50:55.801183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.800748 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" Oct 30 13:50:55.801183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.800770 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" event={"ID":"6042da53-8bfe-484b-84fd-17b27fd80184","Type":"ContainerDied","Data":"ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167"} Oct 30 13:50:55.801183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.800810 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg" event={"ID":"6042da53-8bfe-484b-84fd-17b27fd80184","Type":"ContainerDied","Data":"27ef794d42b8d4e39ec176c9108f2695c8fe425de12e2d5d2762f0b4051fee0c"} Oct 30 13:50:55.801183 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.800833 5693 scope.go:122] "RemoveContainer" containerID="ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167" Oct 30 13:50:55.813496 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.813476 5693 scope.go:122] "RemoveContainer" containerID="ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167" Oct 30 13:50:55.813825 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:50:55.813791 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167\": container with ID starting with ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167 not found: ID does not exist" containerID="ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167" Oct 30 13:50:55.813951 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.813833 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167"} err="failed to get container status \"ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167\": rpc error: code = NotFound desc = could not find container \"ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167\": container with ID starting with ace136ec8826c7f0a6445d973aa5d6a418f317cd8c312390afa0718c8dd5a167 not found: ID does not exist" Oct 30 13:50:55.852190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.852159 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg"] Oct 30 13:50:55.857503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:55.857473 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-5866fjdbg"] Oct 30 13:50:56.191646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:56.191514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:56.277553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:56.277516 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6042da53-8bfe-484b-84fd-17b27fd80184" path="/var/lib/kubelet/pods/6042da53-8bfe-484b-84fd-17b27fd80184/volumes" Oct 30 13:50:56.916884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:56.916851 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:57.742434 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:57.742394 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:59.591445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:50:59.591402 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:00.576581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:00.576531 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:00.712830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:00.712798 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:01.232272 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:01.232242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:01.241018 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:01.240989 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl"] Oct 30 13:51:01.373955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:01.373919 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:01.520080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:01.519984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:03.247969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:03.247933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:04.770378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:04.770344 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:04.903096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:04.903058 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:05.035627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:05.035542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:06.241514 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.241470 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" podUID="f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" containerName="payload-processing" containerID="cri-o://638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f" gracePeriod=25 Oct 30 13:51:06.524311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.524279 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:06.563095 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.563062 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:51:06.608375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.608326 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume\") pod \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " Oct 30 13:51:06.608554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.608468 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume\") pod \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " Oct 30 13:51:06.608732 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.608713 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr\") pod \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " Oct 30 13:51:06.608806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.608756 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials\") pod \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\" (UID: \"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb\") " Oct 30 13:51:06.609915 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.609885 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume" pod "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" (UID: "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:06.610317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.610283 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:51:06.610910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.610886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume" pod "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" (UID: "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:06.613866 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.613829 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials" pod "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" (UID: "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:06.617451 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.617415 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr" pod "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" (UID: "f898d20b-efe1-4dbb-88a4-d14fc45cb9eb"). InnerVolumeSpecName "kube-api-access-mbwmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:06.711543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.711502 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwmr\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-kube-api-access-mbwmr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:51:06.711543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.711536 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:51:06.711543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:06.711547 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:51:07.018583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.018548 5693 generic.go:356] "Generic (PLEG): container finished" podID="f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" containerID="638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f" exitCode=0 Oct 30 13:51:07.018583 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.018607 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" Oct 30 13:51:07.018834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.018632 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" event={"ID":"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb","Type":"ContainerDied","Data":"638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f"} Oct 30 13:51:07.018834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.018707 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl" event={"ID":"f898d20b-efe1-4dbb-88a4-d14fc45cb9eb","Type":"ContainerDied","Data":"439387e7052282b108e058cdb76f24d4c9e02584aaae8e3551d1ef8b5876a0cf"} Oct 30 13:51:07.018834 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.018730 5693 scope.go:122] "RemoveContainer" containerID="638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f" Oct 30 13:51:07.033433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.032517 5693 scope.go:122] "RemoveContainer" containerID="638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f" Oct 30 13:51:07.033911 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:51:07.033874 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f\": container with ID starting with 638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f not found: ID does not exist" containerID="638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f" Oct 30 13:51:07.034033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.033914 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f"} err="failed to get container status \"638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f\": rpc error: code = NotFound desc = could not find container \"638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f\": container with ID starting with 638e7b35b425769ab6cb0ed692c1b2e4bc1cfbdde30aa3ec4bda245decfe156f not found: ID does not exist" Oct 30 13:51:07.071433 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.071404 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl"] Oct 30 13:51:07.083721 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:07.083695 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-5964g5wfl"] Oct 30 13:51:08.076621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:08.076559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:08.227635 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:08.225666 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:08.257894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:08.257863 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" path="/var/lib/kubelet/pods/f898d20b-efe1-4dbb-88a4-d14fc45cb9eb/volumes" Oct 30 13:51:08.511885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:08.511852 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:08.640904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:08.640870 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:09.124897 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.124861 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl"] Oct 30 13:51:09.126245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.126218 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6042da53-8bfe-484b-84fd-17b27fd80184" containerName="payload-processing" Oct 30 13:51:09.126245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.126244 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6042da53-8bfe-484b-84fd-17b27fd80184" containerName="payload-processing" Oct 30 13:51:09.126419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.126265 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" containerName="payload-processing" Oct 30 13:51:09.126419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.126273 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" containerName="payload-processing" Oct 30 13:51:09.126520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.126507 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6042da53-8bfe-484b-84fd-17b27fd80184" containerName="payload-processing" Oct 30 13:51:09.126560 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.126524 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f898d20b-efe1-4dbb-88a4-d14fc45cb9eb" containerName="payload-processing" Oct 30 13:51:09.132705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.132683 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.148878 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.148851 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-926ff0-openshift-default-dockercfg-5g7vs\"" Oct 30 13:51:09.166330 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.166279 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl"] Oct 30 13:51:09.173014 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.172984 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173148 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173225 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2rg\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173332 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173354 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173406 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173517 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173501 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173532 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.173581 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.173553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.219660 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.219586 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:09.275489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.275459 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.275717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.275498 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.275717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.275543 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.275853 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.275739 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.275910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.275860 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.275980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2rg\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276125 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276219 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276170 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276218 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276265 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276694 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276696 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276816 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276696 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.276887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.276830 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.278250 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.278223 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.278758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.278733 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.357640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.354326 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.357640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.354975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2rg\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg\") pod \"e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.387791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.383710 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:09.450620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.449908 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:09.866319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.866287 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl"] Oct 30 13:51:09.867950 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:51:09.867923 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbd7a3b_60f4_4623_8b54_345bff163209.slice/crio-3617cfe623fcfdb28b148190cfd5b38e069d71c0c44b480d57ac9de729d8d1cc WatchSource:0}: Error finding container 3617cfe623fcfdb28b148190cfd5b38e069d71c0c44b480d57ac9de729d8d1cc: Status 404 returned error can't find the container with id 3617cfe623fcfdb28b148190cfd5b38e069d71c0c44b480d57ac9de729d8d1cc Oct 30 13:51:09.871112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.871075 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:51:09.871209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.871147 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:51:09.871209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:09.871176 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:51:10.066335 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.066299 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" event={"ID":"dcbd7a3b-60f4-4623-8b54-345bff163209","Type":"ContainerStarted","Data":"32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f"} Oct 30 13:51:10.066518 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.066363 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" event={"ID":"dcbd7a3b-60f4-4623-8b54-345bff163209","Type":"ContainerStarted","Data":"3617cfe623fcfdb28b148190cfd5b38e069d71c0c44b480d57ac9de729d8d1cc"} Oct 30 13:51:10.110273 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.110225 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podStartSLOduration=1.110213452 podStartE2EDuration="1.110213452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:10.109220818 +0000 UTC m=+2845.975838990" watchObservedRunningTime="2026-10-30 13:51:10.110213452 +0000 UTC m=+2845.976831625" Oct 30 13:51:10.380244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.380195 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx"] Oct 30 13:51:10.385502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.385478 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.407739 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.407710 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-08c365-serving-cert\"" Oct 30 13:51:10.412392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.412359 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx"] Oct 30 13:51:10.450928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.450896 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:10.452916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.452883 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:51:10.453069 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.452958 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:51:10.511980 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.511947 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czspr\" (UniqueName: \"kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.512168 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.512147 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.512342 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.512320 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.512419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.512394 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.535718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.535686 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:10.613850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.613815 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.614046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.613941 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.614046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.613997 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.614152 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.614063 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czspr\" (UniqueName: \"kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.614586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.614560 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.616365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.616334 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.616627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.616574 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.633817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.633751 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czspr\" (UniqueName: \"kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr\") pod \"maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:10.692614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.692550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:10.704021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:10.703978 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:11.037382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:11.037356 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx"] Oct 30 13:51:11.039419 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:51:11.039380 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eaa7daf_31f1_45a7_96cd_3064299f6f6a.slice/crio-5f7aee99855ec5c3f59385925da5ffb711af8f1d5a599bcb430f6eb2c397df31 WatchSource:0}: Error finding container 5f7aee99855ec5c3f59385925da5ffb711af8f1d5a599bcb430f6eb2c397df31: Status 404 returned error can't find the container with id 5f7aee99855ec5c3f59385925da5ffb711af8f1d5a599bcb430f6eb2c397df31 Oct 30 13:51:11.074950 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:11.074911 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" event={"ID":"8eaa7daf-31f1-45a7-96cd-3064299f6f6a","Type":"ContainerStarted","Data":"5f7aee99855ec5c3f59385925da5ffb711af8f1d5a599bcb430f6eb2c397df31"} Oct 30 13:51:11.456131 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:11.456105 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:12.087880 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.087841 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" event={"ID":"8eaa7daf-31f1-45a7-96cd-3064299f6f6a","Type":"ContainerStarted","Data":"db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc"} Oct 30 13:51:12.088918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.088174 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:12.088918 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.088217 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:12.089969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.089946 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:51:12.129694 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.129633 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" podStartSLOduration=1.7734994099999999 podStartE2EDuration="2.129616021s" totalImagesPullingTime="356.116611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:12.127262696 +0000 UTC m=+2847.993880846" watchObservedRunningTime="2026-10-30 13:51:12.129616021 +0000 UTC m=+2847.996234196" Oct 30 13:51:12.640625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.633164 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:12.809313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:12.809282 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:14.493571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:14.493537 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:14.634823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:14.634785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:16.316482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:16.316443 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:17.924628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:17.915938 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:18.102547 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:18.102518 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 13:51:19.525221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:19.525184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:21.212907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:21.212871 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:22.782564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:22.782521 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:23.057152 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:51:23.057070 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:51:24.230727 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.230694 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297"] Oct 30 13:51:24.236447 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.236424 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.254507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.254471 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-88352d-serving-cert\"" Oct 30 13:51:24.261368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.261338 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297"] Oct 30 13:51:24.348550 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.348503 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.348753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.348728 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.348937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.348906 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc87\" (UniqueName: \"kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.349075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.349056 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.354608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.354569 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.450954 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.450922 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.451147 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.451088 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.451221 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.451205 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc87\" (UniqueName: \"kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.451278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.451263 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.451833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.451794 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.453776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.453750 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.454022 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.453992 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.472435 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.472408 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc87\" (UniqueName: \"kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87\") pod \"maas-api-e2e-inf-b-88352d-bb77558c9-mv297\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.506357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.506070 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.557290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.556910 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:24.652645 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.652085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.882180 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:24.882156 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297"] Oct 30 13:51:24.884467 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:51:24.884435 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26296c50_cd6e_4627_891d_705cf55c1b51.slice/crio-673ff5f10a27bc8e1f50c16dbfec914a0a46a56586c2ae0ec7449b901c9ee8ef WatchSource:0}: Error finding container 673ff5f10a27bc8e1f50c16dbfec914a0a46a56586c2ae0ec7449b901c9ee8ef: Status 404 returned error can't find the container with id 673ff5f10a27bc8e1f50c16dbfec914a0a46a56586c2ae0ec7449b901c9ee8ef Oct 30 13:51:25.193267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:25.193179 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" event={"ID":"26296c50-cd6e-4627-891d-705cf55c1b51","Type":"ContainerStarted","Data":"673ff5f10a27bc8e1f50c16dbfec914a0a46a56586c2ae0ec7449b901c9ee8ef"} Oct 30 13:51:26.203506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:26.203473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" event={"ID":"26296c50-cd6e-4627-891d-705cf55c1b51","Type":"ContainerStarted","Data":"7116b796b607373d1a0e7c1d770b2b8f02a349423eb98bca2fb3ea0b0e63e88a"} Oct 30 13:51:26.203941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:26.203678 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:26.268620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:26.268232 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" podStartSLOduration=1.9118964489999999 podStartE2EDuration="2.268212986s" totalImagesPullingTime="356.316537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:26.265994766 +0000 UTC m=+2862.132612989" watchObservedRunningTime="2026-10-30 13:51:26.268212986 +0000 UTC m=+2862.134831160" Oct 30 13:51:26.279625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:26.274252 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:28.057503 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:28.052262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:30.015242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:30.015204 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:32.071511 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:32.071459 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:32.222297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:32.222263 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 13:51:33.865618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:33.864173 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:35.520445 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:35.520405 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:37.285624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:37.280533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:39.036747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:39.033477 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:40.343685 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.343643 5693 generic.go:356] "Generic (PLEG): container finished" podID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerID="15281020e36f866306a87bac0555bc0effee7dceeb9c4d83b7b55de42fc402d6" exitCode=0 Oct 30 13:51:40.344261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.343691 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" event={"ID":"1836005b-46a2-45c3-b269-75bfbf3cabfb","Type":"ContainerDied","Data":"15281020e36f866306a87bac0555bc0effee7dceeb9c4d83b7b55de42fc402d6"} Oct 30 13:51:40.344261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.344161 5693 scope.go:122] "RemoveContainer" containerID="15281020e36f866306a87bac0555bc0effee7dceeb9c4d83b7b55de42fc402d6" Oct 30 13:51:40.346313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.346275 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:51:40.346436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.346355 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:51:40.346436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.346391 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:51:40.711206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:40.711085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:41.362937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:41.362902 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" event={"ID":"1836005b-46a2-45c3-b269-75bfbf3cabfb","Type":"ContainerStarted","Data":"d582f0b0ac880640d6716c4789455b8e25fe10e2801ab91faa39534e530b98e4"} Oct 30 13:51:42.169574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:42.169542 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:51:42.170844 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:42.170815 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:51:42.171003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:42.170872 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:51:42.595941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:42.595901 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:43.170161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:43.170122 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:51:43.170366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:43.170199 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:51:44.170490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:44.170451 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:51:44.170921 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:44.170526 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:51:44.178035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:44.178012 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:51:44.375947 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:44.368554 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:45.170051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:45.170016 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:51:45.170242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:45.170071 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:51:46.172426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:46.172387 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:46.173573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:46.173546 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:51:46.173817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:46.173795 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:51:47.175415 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:47.175382 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:51:47.176535 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:47.176512 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 13:51:48.145127 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:48.145089 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:49.735847 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:49.735814 5693 scope.go:122] "RemoveContainer" containerID="6759e85bb4900672c39d6567690f14ce1705f0fc2fb84fd4cae252c8e4adab03" Oct 30 13:51:49.953696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:49.953654 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:51.641422 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:51.641377 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:53.061668 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:51:53.061592 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:51:53.422667 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:53.422538 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:53.573975 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:53.573930 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:55.180207 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:55.180113 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:56.854261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:56.854230 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:57.046318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:57.046279 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:58.365046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:58.365010 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:58.514165 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:58.514127 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:58.653190 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:51:58.653102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:00.341709 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:00.341672 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:02.015295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:02.015258 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:04.183395 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:04.183312 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:05.989863 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:05.989827 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.436752 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:07.436707 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.664746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:07.664706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.811172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:07.811136 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:08.009944 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:08.009913 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:09.690774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:09.690738 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:09.896290 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:09.896254 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:11.613527 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:11.613486 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:11.829374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:11.829331 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:12.367629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:12.367573 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:12.962625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:12.959748 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:13.430676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:13.430620 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:13.701894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:13.701797 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:15.576812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:15.576774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:17.644077 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:17.644045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:18.079371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:18.079338 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:19.256903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:19.256870 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:19.487153 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:19.487118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:20.158522 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:20.158486 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:21.951646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:21.951583 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:23.056424 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:52:23.056393 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:52:23.649624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:23.648675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:25.326619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:25.326563 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:27.180734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:27.180692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:28.833442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:28.833407 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:30.343456 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:30.343409 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:30.499810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:30.499778 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:31.087671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:31.087641 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:32.200384 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:32.200350 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:34.123836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:34.123805 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:35.308807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:35.308770 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:35.602580 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:35.602491 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:36.018419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:36.018345 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:37.812625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:37.804874 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:39.745621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:39.744184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:41.463340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:41.463249 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:43.095672 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:43.095630 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:43.135676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:43.133779 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:52:43.135676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:43.133870 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:52:44.897719 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:44.897632 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:45.212883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:45.212795 5693 generic.go:356] "Generic (PLEG): container finished" podID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerID="32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f" exitCode=0 Oct 30 13:52:45.212883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:45.212868 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" event={"ID":"dcbd7a3b-60f4-4623-8b54-345bff163209","Type":"ContainerDied","Data":"32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f"} Oct 30 13:52:45.213412 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:45.213385 5693 scope.go:122] "RemoveContainer" containerID="32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f" Oct 30 13:52:45.216051 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:45.216012 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:52:45.216199 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:45.216173 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:52:45.216298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:45.216222 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:52:46.223245 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:46.223206 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" event={"ID":"dcbd7a3b-60f4-4623-8b54-345bff163209","Type":"ContainerStarted","Data":"4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc"} Oct 30 13:52:46.452621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:46.452022 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:52:46.452621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:46.452174 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:52:46.452621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:46.452232 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:52:46.608579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:46.608544 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:47.452623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:47.452023 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:52:47.452623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:47.452102 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:52:48.207118 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:48.207078 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:48.452657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:48.451976 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:52:48.452657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:48.452063 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:52:48.590247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:48.590204 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:49.451766 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:49.451727 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:52:49.452192 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:49.452159 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:52:50.356397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:50.356353 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:50.451982 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:50.451933 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:52:50.452197 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:50.452007 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:52:51.450733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:51.450695 5693 patch_prober.go:29] interesting pod/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:52:51.451169 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:51.450747 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 13:52:52.142927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:52.132798 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:52.455827 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:52.455742 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:52:52.456284 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:52.456251 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:52:52.457108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:52.457085 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 13:52:53.058090 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:52:53.058053 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:52:53.743280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:53.743205 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:54.119962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.119877 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs"] Oct 30 13:52:54.124989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.124962 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.149512 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.149478 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs"] Oct 30 13:52:54.153832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.153804 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-88352d\"/\"openshift-service-ca.crt\"" Oct 30 13:52:54.154068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.153862 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-88352d\"/\"default-dockercfg-t7n5b\"" Oct 30 13:52:54.154068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.153904 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-88352d\"/\"kube-root-ca.crt\"" Oct 30 13:52:54.154265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.153999 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-88352d\"/\"test-model-88352d-kserve-self-signed-certs\"" Oct 30 13:52:54.228230 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.228184 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:54.265360 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.265325 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.265562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.265493 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.265562 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.265555 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lw8\" (UniqueName: \"kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.265733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.265716 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.265849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.265825 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.265994 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.265977 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.374845 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.375052 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.375202 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.375268 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lw8\" (UniqueName: \"kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.375317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.375378 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.376226 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.376823 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.387674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.377395 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.445692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.437137 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.445692 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.439901 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.447508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.447425 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lw8\" (UniqueName: \"kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8\") pod \"test-model-88352d-kserve-6747cfd8dc-stczs\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:54.740175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:54.740144 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:55.050206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:55.050180 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs"] Oct 30 13:52:55.052038 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:52:55.052011 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90bf08af_007f_4e2a_a37d_9bc2493fad0a.slice/crio-c10906116c2d10b1a48ce741c891a689b4e578b53dc5de0aee5fcd54c31db7ea WatchSource:0}: Error finding container c10906116c2d10b1a48ce741c891a689b4e578b53dc5de0aee5fcd54c31db7ea: Status 404 returned error can't find the container with id c10906116c2d10b1a48ce741c891a689b4e578b53dc5de0aee5fcd54c31db7ea Oct 30 13:52:55.319085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:55.318987 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" event={"ID":"90bf08af-007f-4e2a-a37d-9bc2493fad0a","Type":"ContainerStarted","Data":"2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64"} Oct 30 13:52:55.319085 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:55.319034 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" event={"ID":"90bf08af-007f-4e2a-a37d-9bc2493fad0a","Type":"ContainerStarted","Data":"c10906116c2d10b1a48ce741c891a689b4e578b53dc5de0aee5fcd54c31db7ea"} Oct 30 13:52:55.361748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:55.361417 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" podStartSLOduration=1.361398877 podStartE2EDuration="1.361398877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:55.359905661 +0000 UTC m=+2951.226523833" watchObservedRunningTime="2026-10-30 13:52:55.361398877 +0000 UTC m=+2951.228017054" Oct 30 13:52:55.514545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:55.514504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:55.740287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:55.740257 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:52:57.114628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:57.112266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:57.249677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:57.249647 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:58.716146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:52:58.716109 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:00.405997 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:00.405954 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:01.992931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:01.992886 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:03.642346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:03.642307 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:04.740666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:04.740630 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:53:04.753379 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:04.753351 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:53:05.302170 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:05.302131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:05.469017 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:05.468988 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:53:07.135824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:07.135784 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:08.823161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:08.823123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:10.452555 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:10.452519 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:10.596820 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:10.596787 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:10.977417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:10.977378 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:11.463826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:11.463746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:11.653368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:11.653311 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:12.402684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:12.396514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:14.507373 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:14.507315 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:14.661579 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:14.661541 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:14.834374 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:14.834286 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:15.263882 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:15.263835 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:15.483324 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:15.483281 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:16.834669 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:16.827804 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:17.500684 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:17.500647 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:17.648942 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:17.648905 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:18.787045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:18.786949 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:20.617629 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:20.617582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:22.170646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:22.170610 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:23.057683 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:53:23.057644 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:53:23.816851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:23.816809 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:25.379764 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:25.379685 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:26.938358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:26.938325 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:28.745824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:28.745747 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:30.323265 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:30.323224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:31.943455 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:31.943416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:33.635624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:33.634335 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:35.143755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:35.143721 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:36.793956 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:36.793920 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:38.521254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:38.521220 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:40.204836 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:40.204782 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:40.662083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:40.662046 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:40.859967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:40.859931 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:41.295693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:41.295652 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:41.446551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:41.446521 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:41.610735 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:41.606243 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:42.127530 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:42.127492 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:42.129870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:42.129834 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v"] Oct 30 13:53:42.259788 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:42.259759 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:43.771648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:43.771611 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:45.311513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:45.311478 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:46.793321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:46.793288 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n"] Oct 30 13:53:46.793849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:46.793500 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" containerID="cri-o://59d55433673608f2effd21df4dd3b9c906c330b948210cbb884e07621fe571aa" gracePeriod=30 Oct 30 13:53:46.805357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:46.805327 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q"] Oct 30 13:53:46.805608 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:46.805566 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" containerID="cri-o://956bcd977a7cc64c055b341dbd898a604f1310aec7be8c22ec7c2c3030fb653c" gracePeriod=30 Oct 30 13:53:47.105513 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.105415 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:47.130715 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.130678 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" podUID="3e7e6974-db72-4989-a02f-c7887afc9fa7" containerName="payload-processing" containerID="cri-o://2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b" gracePeriod=25 Oct 30 13:53:47.448305 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.448274 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:53:47.521648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.521556 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume\") pod \"3e7e6974-db72-4989-a02f-c7887afc9fa7\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " Oct 30 13:53:47.521648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.521738 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr\") pod \"3e7e6974-db72-4989-a02f-c7887afc9fa7\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " Oct 30 13:53:47.521648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.521832 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials\") pod \"3e7e6974-db72-4989-a02f-c7887afc9fa7\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " Oct 30 13:53:47.522648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.521905 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume\") pod \"3e7e6974-db72-4989-a02f-c7887afc9fa7\" (UID: \"3e7e6974-db72-4989-a02f-c7887afc9fa7\") " Oct 30 13:53:47.522648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.522461 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume" pod "3e7e6974-db72-4989-a02f-c7887afc9fa7" (UID: "3e7e6974-db72-4989-a02f-c7887afc9fa7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:47.522792 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.522705 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-routing-overlay-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:47.522926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.522904 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume" pod "3e7e6974-db72-4989-a02f-c7887afc9fa7" (UID: "3e7e6974-db72-4989-a02f-c7887afc9fa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:47.525013 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.524990 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr" pod "3e7e6974-db72-4989-a02f-c7887afc9fa7" (UID: "3e7e6974-db72-4989-a02f-c7887afc9fa7"). InnerVolumeSpecName "kube-api-access-847tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:47.525123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.525029 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials" pod "3e7e6974-db72-4989-a02f-c7887afc9fa7" (UID: "3e7e6974-db72-4989-a02f-c7887afc9fa7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:47.623865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.623834 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-provider-credentials\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:47.623865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.623862 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e6974-db72-4989-a02f-c7887afc9fa7-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:47.623865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.623872 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847tr\" (UniqueName: \"kubernetes.io/projected/3e7e6974-db72-4989-a02f-c7887afc9fa7-kube-api-access-847tr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:47.934117 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.934085 5693 generic.go:356] "Generic (PLEG): container finished" podID="3e7e6974-db72-4989-a02f-c7887afc9fa7" containerID="2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b" exitCode=0 Oct 30 13:53:47.934545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.934134 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" Oct 30 13:53:47.934545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.934167 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" event={"ID":"3e7e6974-db72-4989-a02f-c7887afc9fa7","Type":"ContainerDied","Data":"2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b"} Oct 30 13:53:47.934545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.934240 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v" event={"ID":"3e7e6974-db72-4989-a02f-c7887afc9fa7","Type":"ContainerDied","Data":"a154b1a83f5f5c6eb8b4821b938459a77a92d8ca0c26d8a31c75f0f0ab7244e5"} Oct 30 13:53:47.934545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.934261 5693 scope.go:122] "RemoveContainer" containerID="2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b" Oct 30 13:53:47.945894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.945877 5693 scope.go:122] "RemoveContainer" containerID="2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b" Oct 30 13:53:47.946170 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:53:47.946150 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b\": container with ID starting with 2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b not found: ID does not exist" containerID="2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b" Oct 30 13:53:47.946223 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.946179 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b"} err="failed to get container status \"2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b\": rpc error: code = NotFound desc = could not find container \"2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b\": container with ID starting with 2cba01162ac8b7d803907b77d64b30afdeaf715516145e9374efcc0dbce1a26b not found: ID does not exist" Oct 30 13:53:47.986492 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.986459 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v"] Oct 30 13:53:47.992902 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:47.992872 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-76f4vjq8v"] Oct 30 13:53:48.257210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:48.257170 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7e6974-db72-4989-a02f-c7887afc9fa7" path="/var/lib/kubelet/pods/3e7e6974-db72-4989-a02f-c7887afc9fa7/volumes" Oct 30 13:53:48.498262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:48.498235 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:48.802210 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:48.802180 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:48.981362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:48.981334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:49.135218 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.135138 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:49.351810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.351770 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb"] Oct 30 13:53:49.353343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.353314 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3e7e6974-db72-4989-a02f-c7887afc9fa7" containerName="payload-processing" Oct 30 13:53:49.353343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.353344 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7e6974-db72-4989-a02f-c7887afc9fa7" containerName="payload-processing" Oct 30 13:53:49.353588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.353572 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7e6974-db72-4989-a02f-c7887afc9fa7" containerName="payload-processing" Oct 30 13:53:49.359708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.359682 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.374501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.374468 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d724c8-openshift-default-dockercfg-pjmpv\"" Oct 30 13:53:49.392913 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.392786 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb"] Oct 30 13:53:49.453690 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.453645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456330 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456380 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456414 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwklg\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456620 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456673 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456814 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456855 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456928 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.457163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.456966 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.563884 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.563963 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564114 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564157 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564233 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564270 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564401 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.564431 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwklg\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.565571 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.565916 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.566531 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.570619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.567438 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.583624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.581390 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.613983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.613944 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.614546 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.614517 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.645790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.628221 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.652561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.651356 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwklg\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg\") pod \"e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.679621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.678853 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:49.704620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:49.702112 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:50.148350 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.148322 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb"] Oct 30 13:53:50.150298 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:53:50.150265 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd851c6f3_ffa9_4084_aa00_dff5be92cb1b.slice/crio-d97c952d143896b944fc64fc59975776a5adbcb38f3e497f099e2ef7cd4ef984 WatchSource:0}: Error finding container d97c952d143896b944fc64fc59975776a5adbcb38f3e497f099e2ef7cd4ef984: Status 404 returned error can't find the container with id d97c952d143896b944fc64fc59975776a5adbcb38f3e497f099e2ef7cd4ef984 Oct 30 13:53:50.153615 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.153565 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:53:50.153738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.153663 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:53:50.153738 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.153712 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:53:50.509444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.508016 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:50.969203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.969114 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" event={"ID":"d851c6f3-ffa9-4084-aa00-dff5be92cb1b","Type":"ContainerStarted","Data":"ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca"} Oct 30 13:53:50.969203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:50.969175 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" event={"ID":"d851c6f3-ffa9-4084-aa00-dff5be92cb1b","Type":"ContainerStarted","Data":"d97c952d143896b944fc64fc59975776a5adbcb38f3e497f099e2ef7cd4ef984"} Oct 30 13:53:51.036892 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.036837 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podStartSLOduration=2.036821218 podStartE2EDuration="2.036821218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:51.022333478 +0000 UTC m=+3006.888951651" watchObservedRunningTime="2026-10-30 13:53:51.036821218 +0000 UTC m=+3006.903439391" Oct 30 13:53:51.680010 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.679974 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:51.686734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.686707 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:51.981573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.980798 5693 generic.go:356] "Generic (PLEG): container finished" podID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerID="956bcd977a7cc64c055b341dbd898a604f1310aec7be8c22ec7c2c3030fb653c" exitCode=0 Oct 30 13:53:51.981573 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.980910 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" event={"ID":"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd","Type":"ContainerDied","Data":"956bcd977a7cc64c055b341dbd898a604f1310aec7be8c22ec7c2c3030fb653c"} Oct 30 13:53:51.982741 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.982700 5693 generic.go:356] "Generic (PLEG): container finished" podID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerID="59d55433673608f2effd21df4dd3b9c906c330b948210cbb884e07621fe571aa" exitCode=0 Oct 30 13:53:51.982826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.982766 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" event={"ID":"1e18e077-bc58-4739-b0e8-1c6117524ece","Type":"ContainerDied","Data":"59d55433673608f2effd21df4dd3b9c906c330b948210cbb884e07621fe571aa"} Oct 30 13:53:51.983103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.983079 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:51.984830 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:51.984806 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:53:52.133313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.133279 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:53:52.164460 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.164429 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:53:52.230416 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230376 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.230665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230422 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.230665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230507 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.230665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230551 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.230665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230584 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.230927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230705 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.230927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230763 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.230927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230814 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.230927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230878 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.230927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230914 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.231266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230805 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.231266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.230998 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.231266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.231266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231174 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.231266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.231568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231302 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token\") pod \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\" (UID: \"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd\") " Oct 30 13:53:52.231568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231400 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:52.231568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231411 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.231568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231499 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.231568 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231557 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert\") pod \"1e18e077-bc58-4739-b0e8-1c6117524ece\" (UID: \"1e18e077-bc58-4739-b0e8-1c6117524ece\") " Oct 30 13:53:52.231894 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.231713 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.232039 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.232015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.232375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.232347 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:52.232644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.232617 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.233115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.233040 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.233878 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "kube-api-access-w5hd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234124 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234147 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5hd8\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-kube-api-access-w5hd8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234164 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234176 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234189 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234201 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234216 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e18e077-bc58-4739-b0e8-1c6117524ece-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.234444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.234228 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.248689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.235101 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.248689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.236806 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.248689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.245104 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:52.248689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.246484 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:52.250313 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.249270 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.251999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.251976 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.252256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.252234 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:53:52.252256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.252231 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:52.254753 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.254727 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo" pod "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" (UID: "a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:53:52.254916 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.254893 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:52.255499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.255472 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv" pod "1e18e077-bc58-4739-b0e8-1c6117524ece" (UID: "1e18e077-bc58-4739-b0e8-1c6117524ece"). InnerVolumeSpecName "kube-api-access-rsftv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:52.336203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336171 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsftv\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-kube-api-access-rsftv\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336202 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336203 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336217 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336234 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336246 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336256 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336267 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336277 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e18e077-bc58-4739-b0e8-1c6117524ece-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336287 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.336483 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.336300 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e18e077-bc58-4739-b0e8-1c6117524ece-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:53:52.992103 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.992066 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" Oct 30 13:53:52.992586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.992078 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q" event={"ID":"a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd","Type":"ContainerDied","Data":"8e2dd037dd0f84e26c565514baf82ea02f16c1d7f1118dcc32fd307cf34e70d2"} Oct 30 13:53:52.992586 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.992198 5693 scope.go:122] "RemoveContainer" containerID="956bcd977a7cc64c055b341dbd898a604f1310aec7be8c22ec7c2c3030fb653c" Oct 30 13:53:52.993795 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.993775 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" Oct 30 13:53:52.993926 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:52.993812 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n" event={"ID":"1e18e077-bc58-4739-b0e8-1c6117524ece","Type":"ContainerDied","Data":"c275d5786c9fa7a044512283d5b35ce7b865f54ea8bda25f23e21a9c732d86c1"} Oct 30 13:53:53.004195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.004169 5693 scope.go:122] "RemoveContainer" containerID="59d55433673608f2effd21df4dd3b9c906c330b948210cbb884e07621fe571aa" Oct 30 13:53:53.057506 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:53:53.057462 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:53:53.064352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.064323 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n"] Oct 30 13:53:53.068045 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.068022 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8g88n"] Oct 30 13:53:53.100959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.100724 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q"] Oct 30 13:53:53.105506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.105483 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v649q"] Oct 30 13:53:53.504870 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.504839 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:53.665407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:53.665373 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:54.162640 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:54.162000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:54.316855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:54.316820 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" path="/var/lib/kubelet/pods/1e18e077-bc58-4739-b0e8-1c6117524ece/volumes" Oct 30 13:53:54.317900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:54.317873 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" path="/var/lib/kubelet/pods/a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd/volumes" Oct 30 13:53:55.720346 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.720308 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:55.787869 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.787834 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq"] Oct 30 13:53:55.788793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.788770 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" Oct 30 13:53:55.788793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.788790 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" Oct 30 13:53:55.788989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.788847 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" Oct 30 13:53:55.788989 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.788853 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" Oct 30 13:53:55.789094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.789000 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e18e077-bc58-4739-b0e8-1c6117524ece" containerName="istio-proxy" Oct 30 13:53:55.789094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.789016 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ec9ec5-eda4-4779-8d24-71f7dd7a7bfd" containerName="istio-proxy" Oct 30 13:53:55.794856 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.794830 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:55.817797 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.817758 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq"] Oct 30 13:53:55.826817 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.826785 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d724c8-dockercfg-qxmnx\"" Oct 30 13:53:55.860934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.860893 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:55.915985 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.915943 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d724c8\"" Oct 30 13:53:55.931171 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.931130 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjdb\" (UniqueName: \"kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb\") pod \"payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:55.931648 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:55.931612 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:56.035823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.035781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjdb\" (UniqueName: \"kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb\") pod \"payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:56.037525 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.037493 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:56.037751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.036590 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:56.049624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.047328 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.087663 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.071834 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjdb\" (UniqueName: \"kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb\") pod \"payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:56.142624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.141873 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:56.729461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:56.729435 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq"] Oct 30 13:53:56.732487 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:53:56.732448 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe33bfc_c4f4_4286_b315_abb0b341bc3e.slice/crio-0f1dec7aaea8102803c24469282f3bdcc15590629c0fab73d40ac9849cd32faa WatchSource:0}: Error finding container 0f1dec7aaea8102803c24469282f3bdcc15590629c0fab73d40ac9849cd32faa: Status 404 returned error can't find the container with id 0f1dec7aaea8102803c24469282f3bdcc15590629c0fab73d40ac9849cd32faa Oct 30 13:53:57.091424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:57.091338 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" event={"ID":"dbe33bfc-c4f4-4286-b315-abb0b341bc3e","Type":"ContainerStarted","Data":"52f5eb0d71680ddbde40e0d843163672826c4652e716e9cf2a22f1e38664d6b1"} Oct 30 13:53:57.091424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:57.091399 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" event={"ID":"dbe33bfc-c4f4-4286-b315-abb0b341bc3e","Type":"ContainerStarted","Data":"0f1dec7aaea8102803c24469282f3bdcc15590629c0fab73d40ac9849cd32faa"} Oct 30 13:53:57.091424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:57.091421 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:53:57.130490 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:57.130442 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" podStartSLOduration=2.130428006 podStartE2EDuration="2.130428006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:57.128205362 +0000 UTC m=+3012.994823535" watchObservedRunningTime="2026-10-30 13:53:57.130428006 +0000 UTC m=+3012.997046179" Oct 30 13:53:57.313848 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:57.313816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:57.792771 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:57.792732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:59.469891 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:53:59.469850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:01.109385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:01.109336 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:02.708523 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:02.708469 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:04.369945 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:04.369905 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:06.024143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:06.024089 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:07.611254 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:07.611220 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:08.150094 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:08.150066 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 13:54:09.223489 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:09.223431 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:10.734389 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.734354 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt"] Oct 30 13:54:10.739487 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.739455 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.766693 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.766660 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-319179-serving-cert\"" Oct 30 13:54:10.773541 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.773507 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt"] Oct 30 13:54:10.835414 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.835381 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gpt\" (UniqueName: \"kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.835622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.835463 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.835622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.835533 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.835622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.835580 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.866122 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.866092 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:10.977622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.963990 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gpt\" (UniqueName: \"kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.977622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.976479 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.977622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.976558 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.977622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.976662 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:10.981104 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:10.981066 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:11.002195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:11.001864 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:11.003841 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:11.003780 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gpt\" (UniqueName: \"kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:11.014371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:11.014337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:11.016938 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:11.016909 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs\") pod \"maas-api-e2e-inf-b-319179-c6bfb87f-nklqt\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:11.054671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:11.054636 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:11.365780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:11.365752 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt"] Oct 30 13:54:11.367356 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:54:11.367328 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb637c852_682d_40f7_8071_987466336178.slice/crio-1b787a799f98a06014a476ad1f1cc2409128fd0b873c590772230b9d5ddddf90 WatchSource:0}: Error finding container 1b787a799f98a06014a476ad1f1cc2409128fd0b873c590772230b9d5ddddf90: Status 404 returned error can't find the container with id 1b787a799f98a06014a476ad1f1cc2409128fd0b873c590772230b9d5ddddf90 Oct 30 13:54:12.305148 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:12.305107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" event={"ID":"b637c852-682d-40f7-8071-987466336178","Type":"ContainerStarted","Data":"13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9"} Oct 30 13:54:12.305565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:12.305167 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" event={"ID":"b637c852-682d-40f7-8071-987466336178","Type":"ContainerStarted","Data":"1b787a799f98a06014a476ad1f1cc2409128fd0b873c590772230b9d5ddddf90"} Oct 30 13:54:12.305565 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:12.305192 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:12.350143 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:12.350095 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" podStartSLOduration=1.9745587420000001 podStartE2EDuration="2.350081934s" totalImagesPullingTime="375.523192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:12.345384893 +0000 UTC m=+3028.212003065" watchObservedRunningTime="2026-10-30 13:54:12.350081934 +0000 UTC m=+3028.216700125" Oct 30 13:54:12.615934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:12.615851 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:14.298297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:14.298266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:15.038976 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:15.038938 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:16.118364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:16.118325 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:16.497331 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:16.497269 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:17.708679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:17.708648 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:18.319925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:18.319898 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 13:54:19.258534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:19.258500 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:21.020630 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:21.020559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:22.913931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:22.913896 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:23.056776 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:54:23.056739 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:54:24.558304 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:24.558271 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:26.201865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:26.201831 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:27.787421 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:27.787374 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:29.525088 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:29.525054 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:31.150873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:31.150835 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:32.749618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:32.728582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:34.406093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:34.406064 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:36.000393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:36.000357 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:37.484285 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:37.484245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:37.633296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:37.633260 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:39.307778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:39.307743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:40.945949 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:40.945890 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:42.514743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:42.514707 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:44.117731 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:44.117696 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:45.647855 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:45.647810 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:47.397558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:47.397526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:48.989927 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:48.989894 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:49.148619 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:49.147614 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:49.313453 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:49.313374 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:49.856746 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:49.856713 5693 scope.go:122] "RemoveContainer" containerID="2ab1a3b5fd4065c281d22ef5d8ef2dd541c0945ad2eb7c493e4b4df89cbf29b4" Oct 30 13:54:50.760673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:50.760631 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:52.510624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:52.508531 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:53.057684 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:54:53.057643 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:54:54.216217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:54.216154 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:55.775388 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:55.775278 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:57.418612 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:57.418562 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:57.556859 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:57.556828 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:59.117484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:54:59.117435 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:00.697097 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:00.697062 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:02.257877 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:02.257842 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:03.946646 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:03.943898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:05.792124 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:05.792089 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:07.819457 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:07.819411 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:09.529031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:09.528996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:11.295714 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:11.295675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:12.816774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:12.816705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:12.960798 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:12.960760 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:14.792480 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:14.792442 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:16.496740 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:16.496706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:16.646130 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:16.646094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:18.090860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:18.090822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:19.787537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:19.787493 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:21.415319 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:21.415274 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:23.057747 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:55:23.057705 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:55:23.091910 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:23.091878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:24.716839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:24.716801 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:26.458639 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:26.455326 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:28.103933 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:28.103902 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:29.709275 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:29.709243 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:31.316337 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:31.316289 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:32.143471 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:32.143427 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz"] Oct 30 13:55:32.143784 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:32.143732 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" containerID="cri-o://c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d" gracePeriod=30 Oct 30 13:55:32.872256 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:32.872224 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:34.561890 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:34.558048 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:36.281053 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:36.281014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:37.428226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.428161 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.598570 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.598658 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.598735 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.598957 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.599060 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.599098 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.599141 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.599192 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.599780 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.599326 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket\") pod \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\" (UID: \"9480a4a3-e090-46a7-abb7-d9174cf8f1b9\") " Oct 30 13:55:37.600452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.600031 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:37.600452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.600232 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:37.600452 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.600370 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:37.605809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.600759 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.605809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.600783 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.605809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.600797 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.605809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.605090 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:55:37.605809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.605414 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:37.605809 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.605412 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "kube-api-access-jqrkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:37.606339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.606314 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:37.608387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.608346 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:37.609644 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.609614 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token" pod "9480a4a3-e090-46a7-abb7-d9174cf8f1b9" (UID: "9480a4a3-e090-46a7-abb7-d9174cf8f1b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:37.702347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.702258 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrkb\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-kube-api-access-jqrkb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.702347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.702300 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.702347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.702310 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.702347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.702319 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.702347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.702327 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.702347 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.702335 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9480a4a3-e090-46a7-abb7-d9174cf8f1b9-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:55:37.890658 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:37.890625 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:38.169000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.168966 5693 generic.go:356] "Generic (PLEG): container finished" podID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerID="c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d" exitCode=0 Oct 30 13:55:38.169198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.169019 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" Oct 30 13:55:38.169198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.169040 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" event={"ID":"9480a4a3-e090-46a7-abb7-d9174cf8f1b9","Type":"ContainerDied","Data":"c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d"} Oct 30 13:55:38.169198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.169112 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz" event={"ID":"9480a4a3-e090-46a7-abb7-d9174cf8f1b9","Type":"ContainerDied","Data":"0f3aa2ac1f6f487bce429fd28448f2aafbd85bf9b590e0a20f2f6bcdde97ee05"} Oct 30 13:55:38.169198 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.169135 5693 scope.go:122] "RemoveContainer" containerID="c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d" Oct 30 13:55:38.181101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.181086 5693 scope.go:122] "RemoveContainer" containerID="9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2" Oct 30 13:55:38.192896 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.192873 5693 scope.go:122] "RemoveContainer" containerID="c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d" Oct 30 13:55:38.193201 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:55:38.193168 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d\": container with ID starting with c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d not found: ID does not exist" containerID="c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d" Oct 30 13:55:38.193281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.193213 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d"} err="failed to get container status \"c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d\": rpc error: code = NotFound desc = could not find container \"c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d\": container with ID starting with c3aa614a0213f0b9a3effae5908adcc54886a84fb90a28c67e78e22424d22a9d not found: ID does not exist" Oct 30 13:55:38.193281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.193231 5693 scope.go:122] "RemoveContainer" containerID="9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2" Oct 30 13:55:38.193525 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:55:38.193500 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2\": container with ID starting with 9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2 not found: ID does not exist" containerID="9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2" Oct 30 13:55:38.193588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.193535 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2"} err="failed to get container status \"9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2\": rpc error: code = NotFound desc = could not find container \"9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2\": container with ID starting with 9b740d8a63d643f5a3f69369b74c33c7ea094e0af29352018d021e5e3a6fb4e2 not found: ID does not exist" Oct 30 13:55:38.221826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.221788 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz"] Oct 30 13:55:38.233292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.233264 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-8mpjz"] Oct 30 13:55:38.257824 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:38.257798 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" path="/var/lib/kubelet/pods/9480a4a3-e090-46a7-abb7-d9174cf8f1b9/volumes" Oct 30 13:55:39.611618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:39.611566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:41.256937 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.250888 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:41.340805 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.340774 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd"] Oct 30 13:55:41.342235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.342204 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" Oct 30 13:55:41.342235 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.342226 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" Oct 30 13:55:41.342466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.342268 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" Oct 30 13:55:41.342466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.342275 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" Oct 30 13:55:41.342466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.342448 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" Oct 30 13:55:41.342466 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.342469 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9480a4a3-e090-46a7-abb7-d9174cf8f1b9" containerName="istio-proxy" Oct 30 13:55:41.348348 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.348323 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.366323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.366292 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd"] Oct 30 13:55:41.372563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.372518 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-319179\"/\"kube-root-ca.crt\"" Oct 30 13:55:41.372563 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.372530 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-319179\"/\"openshift-service-ca.crt\"" Oct 30 13:55:41.377005 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.376980 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-319179\"/\"default-dockercfg-4wdmr\"" Oct 30 13:55:41.377114 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.377047 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-319179\"/\"test-model-319179-kserve-self-signed-certs\"" Oct 30 13:55:41.390801 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.390771 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnb8q\" (UniqueName: \"kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.391246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.390964 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.391246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.391018 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.391246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.391086 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.391246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.391175 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.391485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.391350 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.454638 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.454562 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:41.493135 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493098 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493325 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493295 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnb8q\" (UniqueName: \"kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493369 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493393 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493467 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493437 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493734 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493712 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493822 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493769 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.493862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.493829 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.495574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.495549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.495804 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.495787 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.512186 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.512129 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnb8q\" (UniqueName: \"kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q\") pod \"test-model-319179-kserve-65ff788cf5-vwfxd\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:41.667962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:41.667928 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:42.014393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:42.014319 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd"] Oct 30 13:55:42.016540 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:55:42.016508 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod431841f6_1652_487c_8170_458c16d4251f.slice/crio-01f017b6916b9fec6dd660994bb00c711d75a14527d51ee7c1f65527aff29ee3 WatchSource:0}: Error finding container 01f017b6916b9fec6dd660994bb00c711d75a14527d51ee7c1f65527aff29ee3: Status 404 returned error can't find the container with id 01f017b6916b9fec6dd660994bb00c711d75a14527d51ee7c1f65527aff29ee3 Oct 30 13:55:42.204208 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:42.204172 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" event={"ID":"431841f6-1652-487c-8170-458c16d4251f","Type":"ContainerStarted","Data":"d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491"} Oct 30 13:55:42.204385 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:42.204214 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" event={"ID":"431841f6-1652-487c-8170-458c16d4251f","Type":"ContainerStarted","Data":"01f017b6916b9fec6dd660994bb00c711d75a14527d51ee7c1f65527aff29ee3"} Oct 30 13:55:42.245081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:42.245026 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" podStartSLOduration=1.245009614 podStartE2EDuration="1.245009614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:42.241650989 +0000 UTC m=+3118.108269163" watchObservedRunningTime="2026-10-30 13:55:42.245009614 +0000 UTC m=+3118.111627829" Oct 30 13:55:42.669160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:42.669121 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:42.926711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:42.926628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:44.498316 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:44.498277 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:46.038321 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:46.038275 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:47.846352 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:47.846317 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:49.658751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:49.658713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:51.191195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:51.191161 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:51.505689 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:51.505652 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:51.649620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:51.649531 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:51.668680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:51.668649 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:51.681747 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:51.681716 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:52.199673 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:52.199629 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:52.366620 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:52.365912 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:52.591970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:52.591938 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 13:55:53.078579 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:55:53.078536 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:55:53.565021 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:53.564984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:55.353545 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:55.353504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:56.735035 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:56.735000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:56.903358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:56.903319 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:57.247618 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:57.243165 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:57.540533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:57.540453 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:58.368810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:58.368773 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:58.519431 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:58.519397 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:59.160294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:55:59.160196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:00.822871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:00.822842 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:01.538900 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:01.538868 5693 generic.go:356] "Generic (PLEG): container finished" podID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerID="ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca" exitCode=0 Oct 30 13:56:01.539083 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:01.538941 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" event={"ID":"d851c6f3-ffa9-4084-aa00-dff5be92cb1b","Type":"ContainerDied","Data":"ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca"} Oct 30 13:56:01.539497 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:01.539470 5693 scope.go:122] "RemoveContainer" containerID="ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca" Oct 30 13:56:01.541871 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:01.541839 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:56:01.541972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:01.541913 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:56:01.541972 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:01.541951 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:56:02.419851 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:02.419806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:02.548760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:02.548725 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" event={"ID":"d851c6f3-ffa9-4084-aa00-dff5be92cb1b","Type":"ContainerStarted","Data":"3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67"} Oct 30 13:56:02.679322 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:02.679211 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:56:02.680925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:02.680894 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:56:02.681086 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:02.680956 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 13:56:03.679821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:03.679785 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:56:03.680247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:03.679842 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 13:56:04.122048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:04.122017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:04.679748 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:04.679713 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:56:04.679930 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:04.679773 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 13:56:05.680007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:05.679917 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:56:05.680007 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:05.679989 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 13:56:05.887907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:05.887869 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:06.679252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:06.679215 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:56:06.679440 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:06.679280 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 13:56:06.983246 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:06.983213 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:56:07.552553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:07.552514 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:07.680031 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:07.679989 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:56:07.680216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:07.680054 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 13:56:08.684521 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:08.684490 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:56:08.685534 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:08.685513 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 13:56:09.176628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:09.167339 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:10.702506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:10.702465 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:12.320028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:12.319992 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:13.931773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:13.931717 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:15.487760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:15.487726 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:17.222255 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:17.222218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:18.804191 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:18.804151 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:20.490160 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:20.490123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:22.199868 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:22.199816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:23.057340 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:56:23.057304 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:56:23.805624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:23.798098 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:25.370886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:25.370809 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:27.117862 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:27.117816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:30.659839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:30.659771 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:30.801829 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:30.801790 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz"] Oct 30 13:56:30.802185 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:30.802124 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" containerID="cri-o://37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71" gracePeriod=30 Oct 30 13:56:30.802774 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:30.802747 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:32.061621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:32.061532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:33.769653 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:33.769591 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:33.960884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:33.960849 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg"] Oct 30 13:56:33.961072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:33.961034 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" containerID="cri-o://4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d" gracePeriod=30 Oct 30 13:56:35.318569 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:35.318531 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:36.096473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.096446 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:56:36.258211 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258175 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258237 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258328 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258387 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258372 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258510 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.258666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258573 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258658 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258748 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.258884 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258844 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.259099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.258984 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket\") pod \"0d94421b-6340-457d-a007-3324fc757d63\" (UID: \"0d94421b-6340-457d-a007-3324fc757d63\") " Oct 30 13:56:36.259099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259043 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.259482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259338 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.259482 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259403 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.259833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259808 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.259908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259838 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.259908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259853 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.259908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.259866 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.260121 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.260096 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:36.260571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.260546 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:56:36.260755 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.260735 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "kube-api-access-qw9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:36.260946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.260922 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.261048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.260980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token" pod "0d94421b-6340-457d-a007-3324fc757d63" (UID: "0d94421b-6340-457d-a007-3324fc757d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:36.361407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.361326 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw9kp\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-kube-api-access-qw9kp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.361407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.361355 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d94421b-6340-457d-a007-3324fc757d63-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.361407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.361365 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d94421b-6340-457d-a007-3324fc757d63-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.361407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.361373 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d94421b-6340-457d-a007-3324fc757d63-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.361407 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.361381 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d94421b-6340-457d-a007-3324fc757d63-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.864790 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.864751 5693 generic.go:356] "Generic (PLEG): container finished" podID="0d94421b-6340-457d-a007-3324fc757d63" containerID="37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71" exitCode=0 Oct 30 13:56:36.864977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.864802 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" event={"ID":"0d94421b-6340-457d-a007-3324fc757d63","Type":"ContainerDied","Data":"37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71"} Oct 30 13:56:36.864977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.864868 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" event={"ID":"0d94421b-6340-457d-a007-3324fc757d63","Type":"ContainerDied","Data":"25da15dfd3b6520999f7bf60dd82dd2f7449fc5bf9ddab61b5c92ada314a52fd"} Oct 30 13:56:36.864977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.864892 5693 scope.go:122] "RemoveContainer" containerID="37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71" Oct 30 13:56:36.864977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.864810 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz" Oct 30 13:56:36.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.877384 5693 scope.go:122] "RemoveContainer" containerID="37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71" Oct 30 13:56:36.877717 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:56:36.877696 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71\": container with ID starting with 37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71 not found: ID does not exist" containerID="37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71" Oct 30 13:56:36.877800 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.877724 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71"} err="failed to get container status \"37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71\": rpc error: code = NotFound desc = could not find container \"37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71\": container with ID starting with 37713fef3d8a5efc93e36433807e856d1a5f18cb0fd1c0a8c79237ae535faa71 not found: ID does not exist" Oct 30 13:56:36.917821 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.917791 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz"] Oct 30 13:56:36.930146 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:36.930114 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-r5vfz"] Oct 30 13:56:37.018142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:37.018106 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:38.257158 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:38.257127 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d94421b-6340-457d-a007-3324fc757d63" path="/var/lib/kubelet/pods/0d94421b-6340-457d-a007-3324fc757d63/volumes" Oct 30 13:56:38.614343 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:38.614223 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:39.254931 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.254902 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:56:39.407042 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407002 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407093 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407147 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407192 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407234 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407343 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407503 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:39.407564 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407511 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407656 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:39.407979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407853 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:39.407979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407854 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.407979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:39.407979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.407973 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy\") pod \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\" (UID: \"9cd1fdd1-54e7-4d0e-bd6b-082d21021823\") " Oct 30 13:56:39.408294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.408064 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:39.409368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.409332 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.409368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.409369 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.409592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.409382 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.409592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.409395 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.409592 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.409410 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.409810 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.409751 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:39.410149 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.410124 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "kube-api-access-prg7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:39.410251 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.410151 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:56:39.411281 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.411262 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy" pod "9cd1fdd1-54e7-4d0e-bd6b-082d21021823" (UID: "9cd1fdd1-54e7-4d0e-bd6b-082d21021823"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:39.511236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.511128 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.511236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.511172 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prg7h\" (UniqueName: \"kubernetes.io/projected/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-kube-api-access-prg7h\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.511236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.511187 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.511236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.511201 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd1fdd1-54e7-4d0e-bd6b-082d21021823-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.891289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.891195 5693 generic.go:356] "Generic (PLEG): container finished" podID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerID="4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d" exitCode=0 Oct 30 13:56:39.891289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.891236 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" event={"ID":"9cd1fdd1-54e7-4d0e-bd6b-082d21021823","Type":"ContainerDied","Data":"4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d"} Oct 30 13:56:39.891289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.891272 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" Oct 30 13:56:39.891637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.891342 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" event={"ID":"9cd1fdd1-54e7-4d0e-bd6b-082d21021823","Type":"ContainerDied","Data":"443db123519dfd64a720ebe531e19b2efa5309cb9a26481d2c9a1cabac8cb71c"} Oct 30 13:56:39.891637 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.891376 5693 scope.go:122] "RemoveContainer" containerID="4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d" Oct 30 13:56:39.906366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.906347 5693 scope.go:122] "RemoveContainer" containerID="4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d" Oct 30 13:56:39.906667 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:56:39.906646 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d\": container with ID starting with 4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d not found: ID does not exist" containerID="4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d" Oct 30 13:56:39.906758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.906671 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d"} err="failed to get container status \"4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d\": rpc error: code = NotFound desc = could not find container \"4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d\": container with ID starting with 4c8eb40465d8dbd2bd2d68c9ea92d8b396ca182b4368816e1336ed923329ec8d not found: ID does not exist" Oct 30 13:56:39.956777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.956747 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg"] Oct 30 13:56:39.968242 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:39.968217 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg"] Oct 30 13:56:40.148397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:40.148318 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:56:40.148397 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:40.148367 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-lvmjg" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:56:40.209295 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:40.209263 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:40.274831 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:40.274799 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" path="/var/lib/kubelet/pods/9cd1fdd1-54e7-4d0e-bd6b-082d21021823/volumes" Oct 30 13:56:40.468364 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:40.468280 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:40.608551 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:40.608520 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:41.767003 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:41.766965 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:43.746722 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:43.746689 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:45.190092 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:45.190054 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:46.848622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:46.845790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:47.367202 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:47.367167 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:48.525908 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:48.525874 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:50.200737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:50.200660 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:51.777112 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:51.777080 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:53.058307 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:56:53.058264 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:56:53.413247 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:53.413133 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.838508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.838475 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw"] Oct 30 13:56:54.839315 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.839299 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" Oct 30 13:56:54.839377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.839316 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" Oct 30 13:56:54.839377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.839348 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" Oct 30 13:56:54.839377 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.839354 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" Oct 30 13:56:54.839509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.839499 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d94421b-6340-457d-a007-3324fc757d63" containerName="istio-proxy" Oct 30 13:56:54.839543 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.839518 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd1fdd1-54e7-4d0e-bd6b-082d21021823" containerName="istio-proxy" Oct 30 13:56:54.844886 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.844867 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:54.872282 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.872253 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ca0243-serving-cert\"" Oct 30 13:56:54.879046 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.879017 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw"] Oct 30 13:56:54.965292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.965256 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.965473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.965349 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:54.965473 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.965430 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:54.965652 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.965524 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:54.965812 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:54.965791 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt86c\" (UniqueName: \"kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.067700 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.067662 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.068157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.068128 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.069173 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.069146 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.088791 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.088705 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt86c\" (UniqueName: \"kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.089476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.069084 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.127116 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.104705 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.129666 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.128407 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.131718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.131489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:55.134454 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.134387 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt86c\" (UniqueName: \"kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c\") pod \"maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.159294 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.159260 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:55.466973 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.466948 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw"] Oct 30 13:56:55.468887 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:56:55.468859 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00101a5_da7d_4c82_a16d_67ca567c9d81.slice/crio-09cb42242a2b61372349db83ecad6391c32011a8f52fc006d449ae1550420da0 WatchSource:0}: Error finding container 09cb42242a2b61372349db83ecad6391c32011a8f52fc006d449ae1550420da0: Status 404 returned error can't find the container with id 09cb42242a2b61372349db83ecad6391c32011a8f52fc006d449ae1550420da0 Oct 30 13:56:55.470378 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:55.470355 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:56:56.028941 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:56.028905 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" event={"ID":"d00101a5-da7d-4c82-a16d-67ca567c9d81","Type":"ContainerStarted","Data":"ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646"} Oct 30 13:56:56.029371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:56.028963 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" event={"ID":"d00101a5-da7d-4c82-a16d-67ca567c9d81","Type":"ContainerStarted","Data":"09cb42242a2b61372349db83ecad6391c32011a8f52fc006d449ae1550420da0"} Oct 30 13:56:56.029371 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:56.029217 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:56:56.068382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:56.068336 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" podStartSLOduration=1.714595325 podStartE2EDuration="2.068322592s" totalImagesPullingTime="353.727267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:56.067809272 +0000 UTC m=+3191.934427455" watchObservedRunningTime="2026-10-30 13:56:56.068322592 +0000 UTC m=+3191.934940765" Oct 30 13:56:56.718283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:56.710641 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:57.663540 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:57.663504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:58.423807 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:56:58.419087 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:00.134996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:00.134959 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:01.126182 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:01.126146 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57"] Oct 30 13:57:01.126425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:01.126401 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" containerID="cri-o://2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a" gracePeriod=30 Oct 30 13:57:01.821983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:01.821950 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:02.042392 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:02.042358 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 13:57:03.413649 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:03.413542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:04.930076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:04.930040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:06.417590 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.417565 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:57:06.487671 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.487636 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.487846 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.487684 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.487940 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.487757 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488009 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.487974 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:06.488076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.487982 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:06.488076 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488037 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488077 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488177 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488107 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488277 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488291 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488397 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:06.488501 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488391 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token\") pod \"984445f4-1af1-460d-a436-7d55601d2866\" (UID: \"984445f4-1af1-460d-a436-7d55601d2866\") " Oct 30 13:57:06.488793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488760 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:06.488925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.488785 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:06.489372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.489343 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984445f4-1af1-460d-a436-7d55601d2866-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.489372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.489373 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.489549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.489387 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.489549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.489399 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.489549 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.489412 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.490493 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.490455 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:57:06.490761 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.490738 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "kube-api-access-xdzxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:06.490883 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.490768 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:06.490932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.490902 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token" pod "984445f4-1af1-460d-a436-7d55601d2866" (UID: "984445f4-1af1-460d-a436-7d55601d2866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:06.594536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.594476 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.594536 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.594539 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdzxt\" (UniqueName: \"kubernetes.io/projected/984445f4-1af1-460d-a436-7d55601d2866-kube-api-access-xdzxt\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.595129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.594551 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984445f4-1af1-460d-a436-7d55601d2866-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.595129 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.594561 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984445f4-1af1-460d-a436-7d55601d2866-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:06.699318 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:06.699208 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:07.117718 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.117679 5693 generic.go:356] "Generic (PLEG): container finished" podID="984445f4-1af1-460d-a436-7d55601d2866" containerID="2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a" exitCode=0 Oct 30 13:57:07.117914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.117759 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" Oct 30 13:57:07.117914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.117763 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" event={"ID":"984445f4-1af1-460d-a436-7d55601d2866","Type":"ContainerDied","Data":"2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a"} Oct 30 13:57:07.117914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.117855 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57" event={"ID":"984445f4-1af1-460d-a436-7d55601d2866","Type":"ContainerDied","Data":"defc330e5184b09e25d801e5c6866661c0348a65909da254fdae328d438d35e5"} Oct 30 13:57:07.117914 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.117870 5693 scope.go:122] "RemoveContainer" containerID="2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a" Oct 30 13:57:07.130075 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.130053 5693 scope.go:122] "RemoveContainer" containerID="854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481" Oct 30 13:57:07.139955 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.139934 5693 scope.go:122] "RemoveContainer" containerID="2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a" Oct 30 13:57:07.140246 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:57:07.140222 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a\": container with ID starting with 2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a not found: ID does not exist" containerID="2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a" Oct 30 13:57:07.140311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.140254 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a"} err="failed to get container status \"2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a\": rpc error: code = NotFound desc = could not find container \"2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a\": container with ID starting with 2e7b5e4d9a35544ae3cc8e03a1b8ceaafa9d4c9083b1348162e364c2c1c9dd7a not found: ID does not exist" Oct 30 13:57:07.140311 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.140274 5693 scope.go:122] "RemoveContainer" containerID="854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481" Oct 30 13:57:07.140527 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:57:07.140510 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481\": container with ID starting with 854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481 not found: ID does not exist" containerID="854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481" Oct 30 13:57:07.140571 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.140534 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481"} err="failed to get container status \"854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481\": rpc error: code = NotFound desc = could not find container \"854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481\": container with ID starting with 854c09c29444961b25083d74db6628c884706ac7985992c1e87bf3ef4303a481 not found: ID does not exist" Oct 30 13:57:07.181962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.181926 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57"] Oct 30 13:57:07.193885 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:07.193857 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-s8l57"] Oct 30 13:57:08.256705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:08.256673 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984445f4-1af1-460d-a436-7d55601d2866" path="/var/lib/kubelet/pods/984445f4-1af1-460d-a436-7d55601d2866/volumes" Oct 30 13:57:08.334449 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:08.334411 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:09.962675 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:09.962588 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:11.600226 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:11.600190 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:13.228850 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:13.228815 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:14.865357 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:14.865294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:16.552959 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:16.552920 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:18.234632 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:18.234462 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:19.481929 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:19.481892 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:19.845679 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:19.845061 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:21.397142 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:21.397105 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:22.131261 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:22.131229 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:23.056961 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:57:23.056920 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:57:23.166758 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:23.166717 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:24.804375 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:24.804332 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:26.546339 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:26.546282 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:28.219728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:28.219657 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:29.924622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:29.916177 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:31.610622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:31.607668 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:32.564423 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:32.564383 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh"] Oct 30 13:57:32.564656 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:32.564635 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" containerID="cri-o://2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05" gracePeriod=30 Oct 30 13:57:33.231823 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:33.231792 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:35.004691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:35.004652 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:36.892826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:36.877532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:37.850368 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:37.850296 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:57:38.031484 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031442 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031497 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031560 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031643 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031721 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031772 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031834 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031937 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.031977 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:38.032033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.032037 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.032167 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy\") pod \"91e228f8-06d6-425b-baf5-ee05cc01130c\" (UID: \"91e228f8-06d6-425b-baf5-ee05cc01130c\") " Oct 30 13:57:38.032705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.032182 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:38.032705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.032217 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:38.032705 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.032364 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:38.033071 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.033052 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.033156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.033076 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e228f8-06d6-425b-baf5-ee05cc01130c-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.033156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.033089 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.033156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.033101 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.033156 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.033114 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.034068 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.034043 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:57:38.034184 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.034070 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:38.034258 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.034233 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "kube-api-access-xdqth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:38.034472 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.034454 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy" pod "91e228f8-06d6-425b-baf5-ee05cc01130c" (UID: "91e228f8-06d6-425b-baf5-ee05cc01130c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:38.134228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.134151 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.134228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.134180 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.134228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.134189 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e228f8-06d6-425b-baf5-ee05cc01130c-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.134228 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.134200 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdqth\" (UniqueName: \"kubernetes.io/projected/91e228f8-06d6-425b-baf5-ee05cc01130c-kube-api-access-xdqth\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.415123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.415040 5693 generic.go:356] "Generic (PLEG): container finished" podID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerID="2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05" exitCode=0 Oct 30 13:57:38.415123 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.415097 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" Oct 30 13:57:38.415309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.415103 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" event={"ID":"91e228f8-06d6-425b-baf5-ee05cc01130c","Type":"ContainerDied","Data":"2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05"} Oct 30 13:57:38.415309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.415216 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh" event={"ID":"91e228f8-06d6-425b-baf5-ee05cc01130c","Type":"ContainerDied","Data":"94cf7a6fd0e1b3d561037671bdbe87e95c9c1cc3dd30cbf2140e00d74245083d"} Oct 30 13:57:38.415309 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.415237 5693 scope.go:122] "RemoveContainer" containerID="2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05" Oct 30 13:57:38.431765 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.431743 5693 scope.go:122] "RemoveContainer" containerID="2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05" Oct 30 13:57:38.432008 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:57:38.431986 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05\": container with ID starting with 2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05 not found: ID does not exist" containerID="2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05" Oct 30 13:57:38.432060 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.432017 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05"} err="failed to get container status \"2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05\": rpc error: code = NotFound desc = could not find container \"2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05\": container with ID starting with 2853ce0b7b2dbd726bac0e3aa17eec23b0dfd1ddd0d1a22b8fd1d23128a6be05 not found: ID does not exist" Oct 30 13:57:38.480269 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.480234 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh"] Oct 30 13:57:38.483993 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.483965 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-52qkh"] Oct 30 13:57:38.617903 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:38.617865 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:40.245621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:40.238945 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:40.263832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:40.263797 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" path="/var/lib/kubelet/pods/91e228f8-06d6-425b-baf5-ee05cc01130c/volumes" Oct 30 13:57:41.827654 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:41.827590 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:43.417248 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:43.417214 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:44.700904 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:44.700855 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:45.093839 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:45.093799 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:46.892093 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:46.892025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:47.408925 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:47.408889 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:48.520394 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:48.520345 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:50.037939 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:50.037904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:51.723461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:51.723416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:53.057291 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:57:53.057251 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:57:53.374206 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:53.374115 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:55.013367 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:55.013095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:55.431195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:55.431109 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw"] Oct 30 13:57:55.567150 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:55.567106 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:55.712279 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:55.712157 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:56.799773 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:56.799512 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:57.800962 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:57.800925 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:58.646362 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:58.645131 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:59.648043 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:57:59.648013 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:00.243333 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:00.243293 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:00.431657 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:00.431586 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerName="payload-pre-processing" containerID="cri-o://33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c" gracePeriod=25 Oct 30 13:58:01.951332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:01.951294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:03.530593 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:03.530556 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:03.746554 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:03.746508 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:58:04.438280 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:04.438244 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:04.577860 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:04.577828 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:05.178622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:05.174063 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:06.913163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:06.913128 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:07.811979 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.811944 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq"] Oct 30 13:58:07.813090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813067 5693 cpu_manager.go:414] "containerMap: removing container" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" Oct 30 13:58:07.813090 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813089 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" Oct 30 13:58:07.813267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813116 5693 cpu_manager.go:414] "containerMap: removing container" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" Oct 30 13:58:07.813267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813121 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" Oct 30 13:58:07.813340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813293 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" Oct 30 13:58:07.813340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813305 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" Oct 30 13:58:07.813340 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.813315 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e228f8-06d6-425b-baf5-ee05cc01130c" containerName="istio-proxy" Oct 30 13:58:07.834532 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.834497 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:07.839172 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.839145 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq"] Oct 30 13:58:07.850283 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.850252 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d724c8\"/\"openshift-service-ca.crt\"" Oct 30 13:58:07.850469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.850293 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d724c8\"/\"default-dockercfg-csx4c\"" Oct 30 13:58:07.862358 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.862330 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d724c8\"/\"test-model-d724c8-kserve-self-signed-certs\"" Oct 30 13:58:07.862865 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.862810 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d724c8\"/\"kube-root-ca.crt\"" Oct 30 13:58:07.907887 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.907841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:07.949239 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.949206 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:07.949659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.949285 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxl29\" (UniqueName: \"kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:07.949659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.949418 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:07.949659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.949443 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:07.949659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.949517 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:07.949659 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:07.949535 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.051934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.051887 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.052793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.052766 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxl29\" (UniqueName: \"kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.056708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.052611 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.056889 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.056735 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.057048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.057030 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.057224 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.057206 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.057728 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.057700 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.058372 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.057569 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.058520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.057048 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.083222 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.083133 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxl29\" (UniqueName: \"kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.093967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.093926 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.095674 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.094488 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs\") pod \"test-model-d724c8-kserve-68cb5d6999-2hvcq\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.152776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.152743 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:08.474408 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.474385 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq"] Oct 30 13:58:08.476000 ip-10-0-130-43 kubenswrapper[5693]: W1030 13:58:08.475971 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a546dd6_cef6_4d84_835c_89464b7a63b6.slice/crio-48d78f1a5e42304181672bae350241b4cdc1577914cbcf5601c5aacf61a26800 WatchSource:0}: Error finding container 48d78f1a5e42304181672bae350241b4cdc1577914cbcf5601c5aacf61a26800: Status 404 returned error can't find the container with id 48d78f1a5e42304181672bae350241b4cdc1577914cbcf5601c5aacf61a26800 Oct 30 13:58:08.519999 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.519921 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:08.702262 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.702225 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" event={"ID":"6a546dd6-cef6-4d84-835c-89464b7a63b6","Type":"ContainerStarted","Data":"2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051"} Oct 30 13:58:08.702444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.702270 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" event={"ID":"6a546dd6-cef6-4d84-835c-89464b7a63b6","Type":"ContainerStarted","Data":"48d78f1a5e42304181672bae350241b4cdc1577914cbcf5601c5aacf61a26800"} Oct 30 13:58:08.746240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:08.746187 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" podStartSLOduration=1.7461729049999999 podStartE2EDuration="1.746172905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:08.743098881 +0000 UTC m=+3264.609717054" watchObservedRunningTime="2026-10-30 13:58:08.746172905 +0000 UTC m=+3264.612791077" Oct 30 13:58:09.153081 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:09.152999 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:10.169461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:10.169404 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:11.750491 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:11.749206 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:13.408787 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:13.408745 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:13.745651 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:13.745590 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:58:15.073213 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:15.073177 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:16.693155 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:16.693115 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:16.855041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:16.849438 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:18.153234 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:18.153194 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:18.165806 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:18.165779 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:18.673628 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:18.664767 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:18.870101 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:18.870068 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 13:58:20.432436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.432404 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:20.732195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.732161 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:58:20.766072 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.766041 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume\") pod \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " Oct 30 13:58:20.766216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.766160 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl\" (UniqueName: \"kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl\") pod \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\" (UID: \"bc314fbc-93c7-453b-9dfc-1566797dbfa6\") " Oct 30 13:58:20.766478 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.766456 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume" pod "bc314fbc-93c7-453b-9dfc-1566797dbfa6" (UID: "bc314fbc-93c7-453b-9dfc-1566797dbfa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:58:20.766909 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.766892 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc314fbc-93c7-453b-9dfc-1566797dbfa6-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:20.768189 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.768169 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl" pod "bc314fbc-93c7-453b-9dfc-1566797dbfa6" (UID: "bc314fbc-93c7-453b-9dfc-1566797dbfa6"). InnerVolumeSpecName "kube-api-access-s6mzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:58:20.868524 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.868489 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mzl\" (UniqueName: \"kubernetes.io/projected/bc314fbc-93c7-453b-9dfc-1566797dbfa6-kube-api-access-s6mzl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:20.877749 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.877720 5693 generic.go:356] "Generic (PLEG): container finished" podID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerID="33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c" exitCode=0 Oct 30 13:58:20.877893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.877765 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" Oct 30 13:58:20.877893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.877807 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" event={"ID":"bc314fbc-93c7-453b-9dfc-1566797dbfa6","Type":"ContainerDied","Data":"33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c"} Oct 30 13:58:20.877893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.877864 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw" event={"ID":"bc314fbc-93c7-453b-9dfc-1566797dbfa6","Type":"ContainerDied","Data":"fb11514a2df294e694f732bd922254b97c78b7344e6c51aba8bcf47a10202248"} Oct 30 13:58:20.877893 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.877889 5693 scope.go:122] "RemoveContainer" containerID="33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c" Oct 30 13:58:20.890078 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.890060 5693 scope.go:122] "RemoveContainer" containerID="33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c" Oct 30 13:58:20.890367 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:58:20.890348 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c\": container with ID starting with 33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c not found: ID does not exist" containerID="33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c" Oct 30 13:58:20.890426 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.890374 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c"} err="failed to get container status \"33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c\": rpc error: code = NotFound desc = could not find container \"33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c\": container with ID starting with 33fb5f3ca559a1fb10ff6f04229f061093a9a6610bad84e2c4da3a26b16daf5c not found: ID does not exist" Oct 30 13:58:20.927969 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.927938 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw"] Oct 30 13:58:20.933624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:20.933580 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5494c9-d5b597d79-ngvgw"] Oct 30 13:58:22.212080 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:22.212045 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:22.257287 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:22.257253 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" path="/var/lib/kubelet/pods/bc314fbc-93c7-453b-9dfc-1566797dbfa6/volumes" Oct 30 13:58:23.058142 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:58:23.058104 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:58:23.839622 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:23.827558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:24.542057 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:24.542027 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:25.411537 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:25.411500 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:27.078041 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:27.078005 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:28.753391 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:28.743681 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:30.346351 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:30.345040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:32.009520 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:32.009487 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:33.704459 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:33.704425 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:35.282770 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:35.282732 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:36.926187 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:36.923408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:38.569717 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:38.569681 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:38.780854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:38.780819 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs"] Oct 30 13:58:38.890327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:38.890237 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:40.178096 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:40.178053 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:40.784244 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:40.784208 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:41.012301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:41.012260 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:41.155743 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:41.154679 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:41.584670 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:41.584635 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:42.832745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:42.832714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:44.469062 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:44.467727 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:46.186008 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:46.185975 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:47.800730 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:47.800686 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:48.606873 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:48.606838 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:48.761439 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:48.761348 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:49.039706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:49.030440 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:49.723089 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:49.723048 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:49.975026 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:49.974485 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2"] Oct 30 13:58:50.140298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:50.140259 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:50.286181 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:50.286100 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:51.568355 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:51.561701 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:53.057300 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:58:53.057254 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:58:53.234000 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:53.233967 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:53.803099 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:53.803057 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" podUID="90bf08af-007f-4e2a-a37d-9bc2493fad0a" containerName="main" containerID="cri-o://2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64" gracePeriod=45 Oct 30 13:58:54.091267 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.091238 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:58:54.142249 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142218 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8\" (UniqueName: \"kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8\") pod \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " Oct 30 13:58:54.142417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142284 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs\") pod \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " Oct 30 13:58:54.142417 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142350 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir\") pod \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " Oct 30 13:58:54.142507 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142452 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache\") pod \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " Oct 30 13:58:54.142558 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142524 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm\") pod \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " Oct 30 13:58:54.142647 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home\") pod \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\" (UID: \"90bf08af-007f-4e2a-a37d-9bc2493fad0a\") " Oct 30 13:58:54.142723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142681 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir" pod "90bf08af-007f-4e2a-a37d-9bc2493fad0a" (UID: "90bf08af-007f-4e2a-a37d-9bc2493fad0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:58:54.142832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142809 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache" pod "90bf08af-007f-4e2a-a37d-9bc2493fad0a" (UID: "90bf08af-007f-4e2a-a37d-9bc2493fad0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:58:54.142977 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.142954 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home" pod "90bf08af-007f-4e2a-a37d-9bc2493fad0a" (UID: "90bf08af-007f-4e2a-a37d-9bc2493fad0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:58:54.143444 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.143423 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:54.143508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.143452 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:54.143508 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.143476 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:54.144983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.144958 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8" pod "90bf08af-007f-4e2a-a37d-9bc2493fad0a" (UID: "90bf08af-007f-4e2a-a37d-9bc2493fad0a"). InnerVolumeSpecName "kube-api-access-b6lw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:58:54.145140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.145104 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm" pod "90bf08af-007f-4e2a-a37d-9bc2493fad0a" (UID: "90bf08af-007f-4e2a-a37d-9bc2493fad0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:58:54.145502 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.145480 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs" pod "90bf08af-007f-4e2a-a37d-9bc2493fad0a" (UID: "90bf08af-007f-4e2a-a37d-9bc2493fad0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:58:54.191499 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.191468 5693 generic.go:356] "Generic (PLEG): container finished" podID="90bf08af-007f-4e2a-a37d-9bc2493fad0a" containerID="2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64" exitCode=0 Oct 30 13:58:54.191697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.191521 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" Oct 30 13:58:54.191697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.191553 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" event={"ID":"90bf08af-007f-4e2a-a37d-9bc2493fad0a","Type":"ContainerDied","Data":"2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64"} Oct 30 13:58:54.191697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.191632 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs" event={"ID":"90bf08af-007f-4e2a-a37d-9bc2493fad0a","Type":"ContainerDied","Data":"c10906116c2d10b1a48ce741c891a689b4e578b53dc5de0aee5fcd54c31db7ea"} Oct 30 13:58:54.191697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.191653 5693 scope.go:122] "RemoveContainer" containerID="2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64" Oct 30 13:58:54.205691 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.205669 5693 scope.go:122] "RemoveContainer" containerID="2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64" Oct 30 13:58:54.206217 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:58:54.206149 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64\": container with ID starting with 2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64 not found: ID does not exist" containerID="2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64" Oct 30 13:58:54.206217 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.206189 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64"} err="failed to get container status \"2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64\": rpc error: code = NotFound desc = could not find container \"2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64\": container with ID starting with 2d0210e946572daaf4df6eb01bf85e0a32c7da100cfa764e0c443384e2276f64 not found: ID does not exist" Oct 30 13:58:54.243928 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.243893 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs"] Oct 30 13:58:54.245084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.245059 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bf08af-007f-4e2a-a37d-9bc2493fad0a-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:54.245084 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.245085 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6lw8\" (UniqueName: \"kubernetes.io/projected/90bf08af-007f-4e2a-a37d-9bc2493fad0a-kube-api-access-b6lw8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:54.245270 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.245094 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bf08af-007f-4e2a-a37d-9bc2493fad0a-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:58:54.255697 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.255677 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-88352d/test-model-88352d-kserve-6747cfd8dc-stczs"] Oct 30 13:58:54.899301 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.895972 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:54.975225 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:54.975186 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerName="payload-processing" containerID="cri-o://5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29" gracePeriod=25 Oct 30 13:58:56.255584 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:56.255551 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bf08af-007f-4e2a-a37d-9bc2493fad0a" path="/var/lib/kubelet/pods/90bf08af-007f-4e2a-a37d-9bc2493fad0a/volumes" Oct 30 13:58:56.461033 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:56.460935 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:57.920837 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:57.920806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:58.072163 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:58.072115 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:59.716627 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:59.713132 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:59.893297 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:58:59.893263 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:59:00.089243 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:00.089210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:00.237429 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:00.237375 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:01.349436 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:01.349383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:02.446626 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:02.446564 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:02.955625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:02.936347 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:04.259642 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.259565 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg"] Oct 30 13:59:04.260141 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.259900 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" podUID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" containerName="maas-api" containerID="cri-o://1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9" gracePeriod=30 Oct 30 13:59:04.342706 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.342679 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:04.392500 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.392467 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" podUID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.255:8443/health\": dial tcp 10.134.0.255:8443: connect: connection refused" Oct 30 13:59:04.780826 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.780793 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:59:04.951946 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.951913 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs\") pod \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " Oct 30 13:59:04.952157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.951986 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx\" (UniqueName: \"kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx\") pod \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " Oct 30 13:59:04.952157 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.952088 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls\") pod \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " Oct 30 13:59:04.952298 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.952210 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca\") pod \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\" (UID: \"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18\") " Oct 30 13:59:04.952588 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.952564 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca" pod "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" (UID: "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:59:04.952983 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.952965 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:04.954366 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.954342 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls" pod "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" (UID: "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:59:04.954708 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.954686 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx" pod "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" (UID: "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18"). InnerVolumeSpecName "kube-api-access-xhcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:59:04.954783 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.954702 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs" pod "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" (UID: "d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:59:04.999424 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:04.999396 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:05.054516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.054481 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:05.054516 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.054515 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:05.054751 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.054525 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhcdx\" (UniqueName: \"kubernetes.io/projected/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18-kube-api-access-xhcdx\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:05.177833 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.177800 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:05.289760 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.289727 5693 generic.go:356] "Generic (PLEG): container finished" podID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" containerID="1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9" exitCode=0 Oct 30 13:59:05.290240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.289777 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" Oct 30 13:59:05.290240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.289808 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" event={"ID":"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18","Type":"ContainerDied","Data":"1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9"} Oct 30 13:59:05.290240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.289862 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg" event={"ID":"d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18","Type":"ContainerDied","Data":"bab988fc789ab0583c2306e92137ff02f9d41fde79f4e83f08d7a929e4290fb5"} Oct 30 13:59:05.290240 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.289883 5693 scope.go:122] "RemoveContainer" containerID="1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9" Oct 30 13:59:05.302419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.302400 5693 scope.go:122] "RemoveContainer" containerID="1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9" Oct 30 13:59:05.302786 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:59:05.302763 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9\": container with ID starting with 1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9 not found: ID does not exist" containerID="1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9" Oct 30 13:59:05.302872 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.302793 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9"} err="failed to get container status \"1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9\": rpc error: code = NotFound desc = could not find container \"1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9\": container with ID starting with 1d3dc03c66294110f39f82caad4edeced335058515ffbc0091371eeb35b5bbd9 not found: ID does not exist" Oct 30 13:59:05.342065 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.342039 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg"] Oct 30 13:59:05.353082 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.353055 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51605e-7dcbd7956d-sxhbg"] Oct 30 13:59:05.612737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:05.612645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:06.256425 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:06.256393 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" path="/var/lib/kubelet/pods/d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18/volumes" Oct 30 13:59:06.826625 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:06.825432 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:08.472175 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:08.472139 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:09.892419 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:09.892384 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:59:10.039907 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:10.039870 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:11.849028 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:11.848992 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:13.467665 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:13.467626 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:13.618289 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:13.618253 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:13.833711 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:13.833681 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:15.273469 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.273443 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:59:15.278754 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.278730 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:15.322723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.319734 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x\" (UniqueName: \"kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x\") pod \"b842c40d-bab5-47fe-b667-f718da5cfa3b\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " Oct 30 13:59:15.322723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.319930 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume\") pod \"b842c40d-bab5-47fe-b667-f718da5cfa3b\" (UID: \"b842c40d-bab5-47fe-b667-f718da5cfa3b\") " Oct 30 13:59:15.322723 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.321739 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume" pod "b842c40d-bab5-47fe-b667-f718da5cfa3b" (UID: "b842c40d-bab5-47fe-b667-f718da5cfa3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:59:15.335418 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.335382 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x" pod "b842c40d-bab5-47fe-b667-f718da5cfa3b" (UID: "b842c40d-bab5-47fe-b667-f718da5cfa3b"). InnerVolumeSpecName "kube-api-access-srg2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:59:15.373776 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.373741 5693 generic.go:356] "Generic (PLEG): container finished" podID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerID="5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29" exitCode=0 Oct 30 13:59:15.373970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.373798 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" Oct 30 13:59:15.373970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.373825 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" event={"ID":"b842c40d-bab5-47fe-b667-f718da5cfa3b","Type":"ContainerDied","Data":"5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29"} Oct 30 13:59:15.373970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.373886 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2" event={"ID":"b842c40d-bab5-47fe-b667-f718da5cfa3b","Type":"ContainerDied","Data":"377dbb4ec417116f9b1d32284c07121e887694b81ae9dfb608615abf341c4ab1"} Oct 30 13:59:15.373970 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.373916 5693 scope.go:122] "RemoveContainer" containerID="5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29" Oct 30 13:59:15.385151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.385130 5693 scope.go:122] "RemoveContainer" containerID="5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29" Oct 30 13:59:15.385405 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:59:15.385384 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29\": container with ID starting with 5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29 not found: ID does not exist" containerID="5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29" Oct 30 13:59:15.385476 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.385416 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29"} err="failed to get container status \"5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29\": rpc error: code = NotFound desc = could not find container \"5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29\": container with ID starting with 5e00d0500db208a5454fe11a5a09263d178b52f8c5835a3e86e4aa3052a52b29 not found: ID does not exist" Oct 30 13:59:15.422393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.422360 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srg2x\" (UniqueName: \"kubernetes.io/projected/b842c40d-bab5-47fe-b667-f718da5cfa3b-kube-api-access-srg2x\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:15.422393 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.422386 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b842c40d-bab5-47fe-b667-f718da5cfa3b-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:15.424509 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.424491 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2"] Oct 30 13:59:15.428317 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:15.428281 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51605e-655fcd5584-w6bv2"] Oct 30 13:59:16.174553 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:16.174510 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:16.256932 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:16.256900 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" path="/var/lib/kubelet/pods/b842c40d-bab5-47fe-b667-f718da5cfa3b/volumes" Oct 30 13:59:17.143174 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:17.143130 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:18.798236 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:18.798124 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:20.576161 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:20.574193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:22.231442 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:22.226586 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:23.056250 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:59:23.056211 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:59:23.776967 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:23.776933 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:24.722854 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:24.722811 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22"] Oct 30 13:59:24.748106 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:24.748075 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm"] Oct 30 13:59:24.878943 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:24.878909 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:25.026252 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:25.026218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:26.554737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:26.554697 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:27.100120 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:27.100088 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:28.732777 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:28.732742 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:29.723745 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:29.723685 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerName="payload-processing" containerID="cri-o://4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c" gracePeriod=25 Oct 30 13:59:29.748832 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:29.748775 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerName="payload-pre-processing" containerID="cri-o://97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57" gracePeriod=25 Oct 30 13:59:30.504140 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:30.504107 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:32.121485 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:32.121453 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:33.860733 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:33.860698 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:35.655621 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:35.645185 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:36.560108 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:36.560066 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:59:37.314623 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:37.312640 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:37.568376 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:37.568296 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:59:38.925151 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:38.925118 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:40.611533 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:40.611491 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:42.233802 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:42.224689 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:43.901843 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:43.901780 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:45.530314 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:45.530277 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:46.560195 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:46.560158 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:59:46.804934 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:46.804887 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:47.492110 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:47.492035 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:47.568849 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:47.568811 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:59:48.921971 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:48.921936 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:49.533209 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:49.533142 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:49.697327 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:49.697287 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:50.054067 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.054048 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:59:50.080680 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.080655 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:59:50.150461 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.150425 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4\" (UniqueName: \"kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4\") pod \"daa12f47-cfae-4e2c-afba-686ea76e6b59\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " Oct 30 13:59:50.150664 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.150644 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume\") pod \"a349a68f-51c7-4efa-bee6-6d1517db24dd\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " Oct 30 13:59:50.150737 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.150713 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f\" (UniqueName: \"kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f\") pod \"a349a68f-51c7-4efa-bee6-6d1517db24dd\" (UID: \"a349a68f-51c7-4efa-bee6-6d1517db24dd\") " Oct 30 13:59:50.150888 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.150868 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume\") pod \"daa12f47-cfae-4e2c-afba-686ea76e6b59\" (UID: \"daa12f47-cfae-4e2c-afba-686ea76e6b59\") " Oct 30 13:59:50.151382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.151082 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume" pod "a349a68f-51c7-4efa-bee6-6d1517db24dd" (UID: "a349a68f-51c7-4efa-bee6-6d1517db24dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:59:50.151382 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.151273 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume" pod "daa12f47-cfae-4e2c-afba-686ea76e6b59" (UID: "daa12f47-cfae-4e2c-afba-686ea76e6b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:59:50.151676 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.151652 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a349a68f-51c7-4efa-bee6-6d1517db24dd-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:50.151781 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.151679 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa12f47-cfae-4e2c-afba-686ea76e6b59-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:50.153266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.153246 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4" pod "daa12f47-cfae-4e2c-afba-686ea76e6b59" (UID: "daa12f47-cfae-4e2c-afba-686ea76e6b59"). InnerVolumeSpecName "kube-api-access-c7pb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:59:50.154609 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.154576 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f" pod "a349a68f-51c7-4efa-bee6-6d1517db24dd" (UID: "a349a68f-51c7-4efa-bee6-6d1517db24dd"). InnerVolumeSpecName "kube-api-access-q2n2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:59:50.256365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.256326 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2n2f\" (UniqueName: \"kubernetes.io/projected/a349a68f-51c7-4efa-bee6-6d1517db24dd-kube-api-access-q2n2f\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:50.256365 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.256363 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pb4\" (UniqueName: \"kubernetes.io/projected/daa12f47-cfae-4e2c-afba-686ea76e6b59-kube-api-access-c7pb4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 13:59:50.317506 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.317407 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:50.704381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.704296 5693 generic.go:356] "Generic (PLEG): container finished" podID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerID="4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c" exitCode=0 Oct 30 13:59:50.704381 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.704355 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" Oct 30 13:59:50.704594 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.704384 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" event={"ID":"daa12f47-cfae-4e2c-afba-686ea76e6b59","Type":"ContainerDied","Data":"4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c"} Oct 30 13:59:50.704594 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.704437 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22" event={"ID":"daa12f47-cfae-4e2c-afba-686ea76e6b59","Type":"ContainerDied","Data":"8e63a0c916446bdfb36ecdcd64ae579f47e6194953937a71ca9848047976f954"} Oct 30 13:59:50.704594 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.704464 5693 scope.go:122] "RemoveContainer" containerID="4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c" Oct 30 13:59:50.708614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.706079 5693 generic.go:356] "Generic (PLEG): container finished" podID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerID="97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57" exitCode=0 Oct 30 13:59:50.708614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.706114 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" event={"ID":"a349a68f-51c7-4efa-bee6-6d1517db24dd","Type":"ContainerDied","Data":"97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57"} Oct 30 13:59:50.708614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.706145 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" Oct 30 13:59:50.708614 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.706153 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm" event={"ID":"a349a68f-51c7-4efa-bee6-6d1517db24dd","Type":"ContainerDied","Data":"cedb913106d1cb3f91b38ea3abef1fa7a910ed85ee7d4f3a407234d10e198069"} Oct 30 13:59:50.716996 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.716976 5693 scope.go:122] "RemoveContainer" containerID="4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c" Oct 30 13:59:50.717292 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:59:50.717275 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c\": container with ID starting with 4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c not found: ID does not exist" containerID="4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c" Oct 30 13:59:50.717336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.717301 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c"} err="failed to get container status \"4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c\": rpc error: code = NotFound desc = could not find container \"4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c\": container with ID starting with 4eff5df20a1a84795081fd1ccce6b9f7bbb95fce19c005cd3300caa65b15a20c not found: ID does not exist" Oct 30 13:59:50.717336 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.717317 5693 scope.go:122] "RemoveContainer" containerID="97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57" Oct 30 13:59:50.727302 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.727284 5693 scope.go:122] "RemoveContainer" containerID="97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57" Oct 30 13:59:50.727565 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:59:50.727543 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57\": container with ID starting with 97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57 not found: ID does not exist" containerID="97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57" Oct 30 13:59:50.727687 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.727573 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57"} err="failed to get container status \"97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57\": rpc error: code = NotFound desc = could not find container \"97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57\": container with ID starting with 97d35ab07866328966be2afc8d4ac09a038c3aebcca2d48f8164c80fabcc9f57 not found: ID does not exist" Oct 30 13:59:50.756048 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.756018 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22"] Oct 30 13:59:50.767574 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.767549 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88352d-55f57586cd-sbj22"] Oct 30 13:59:50.792323 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.792293 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm"] Oct 30 13:59:50.796126 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:50.796093 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88352d-5fc495c9ff-s85rm"] Oct 30 13:59:51.594292 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:51.590489 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:52.255778 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:52.255744 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" path="/var/lib/kubelet/pods/a349a68f-51c7-4efa-bee6-6d1517db24dd/volumes" Oct 30 13:59:52.256332 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:52.256309 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" path="/var/lib/kubelet/pods/daa12f47-cfae-4e2c-afba-686ea76e6b59/volumes" Oct 30 13:59:53.055901 ip-10-0-130-43 kubenswrapper[5693]: E1030 13:59:53.055863 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 13:59:53.194624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:53.189637 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:54.945216 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:54.942331 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:56.612561 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:56.602987 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:57.243011 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:57.242974 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:57.403624 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:57.398199 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:57.554696 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:57.538165 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs"] Oct 30 13:59:57.555481 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:57.555434 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" containerID="cri-o://d582f0b0ac880640d6716c4789455b8e25fe10e2801ab91faa39534e530b98e4" gracePeriod=30 Oct 30 13:59:57.589115 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:57.589090 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:58.058296 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:58.058254 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:58.464677 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:58.461559 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:58.626793 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:58.626758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:58.801266 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:58.801237 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:59.399278 ip-10-0-130-43 kubenswrapper[5693]: I1030 13:59:59.399234 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:00.278587 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.278541 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:00.281074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.281046 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85"] Oct 30 14:00:00.282956 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.282929 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerName="payload-processing" Oct 30 14:00:00.282956 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.282956 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerName="payload-processing" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.282994 5693 cpu_manager.go:414] "containerMap: removing container" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283003 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984445f4-1af1-460d-a436-7d55601d2866" containerName="istio-proxy" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283030 5693 cpu_manager.go:414] "containerMap: removing container" podUID="90bf08af-007f-4e2a-a37d-9bc2493fad0a" containerName="main" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283038 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bf08af-007f-4e2a-a37d-9bc2493fad0a" containerName="main" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283073 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" containerName="maas-api" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283081 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" containerName="maas-api" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283103 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerName="payload-pre-processing" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283111 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerName="payload-pre-processing" Oct 30 14:00:00.283116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283121 5693 cpu_manager.go:414] "containerMap: removing container" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerName="payload-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283125 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerName="payload-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283140 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerName="payload-pre-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283145 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerName="payload-pre-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283304 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b842c40d-bab5-47fe-b667-f718da5cfa3b" containerName="payload-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283318 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bf08af-007f-4e2a-a37d-9bc2493fad0a" containerName="main" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283327 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f86acb-0da5-4154-a4f8-c8dfbbb0ea18" containerName="maas-api" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283336 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc314fbc-93c7-453b-9dfc-1566797dbfa6" containerName="payload-pre-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283341 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa12f47-cfae-4e2c-afba-686ea76e6b59" containerName="payload-processing" Oct 30 14:00:00.284074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.283348 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a349a68f-51c7-4efa-bee6-6d1517db24dd" containerName="payload-pre-processing" Oct 30 14:00:00.289739 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.289716 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67"] Oct 30 14:00:00.290801 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.290779 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.295188 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.295169 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.311859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.311829 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85"] Oct 30 14:00:00.321455 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.321427 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k"] Oct 30 14:00:00.325341 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.325311 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\"" Oct 30 14:00:00.327578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.327551 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67"] Oct 30 14:00:00.327751 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.327705 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.337291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.337260 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh"] Oct 30 14:00:00.341940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.341915 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.355164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.355133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.355384 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.355270 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwcj\" (UniqueName: \"kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.355512 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.355435 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.355645 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.355518 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ktp\" (UniqueName: \"kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.377621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.377573 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k"] Oct 30 14:00:00.384775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.384738 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl"] Oct 30 14:00:00.415941 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.415908 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh"] Oct 30 14:00:00.416423 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.415976 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w"] Oct 30 14:00:00.416956 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.416930 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.424748 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.424684 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.451502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.451469 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl"] Oct 30 14:00:00.457227 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457187 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwcj\" (UniqueName: \"kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.457390 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457260 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hbk\" (UniqueName: \"kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.457390 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbzt\" (UniqueName: \"kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.457519 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457410 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.457519 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457506 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ktp\" (UniqueName: \"kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.457663 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457560 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9c8w\" (UniqueName: \"kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.457985 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457698 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.457985 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457799 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.457985 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.457916 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.458187 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.458076 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.459358 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.459295 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.459886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.459864 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.468575 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.467979 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w"] Oct 30 14:00:00.486225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.486189 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ktp\" (UniqueName: \"kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.487844 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.487813 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwcj\" (UniqueName: \"kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.559664 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559567 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9c8w\" (UniqueName: \"kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.559664 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559634 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.559954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559684 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.559954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559754 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.559954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559852 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hbk\" (UniqueName: \"kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.559954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559872 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z62k\" (UniqueName: \"kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.559954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.559924 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbzt\" (UniqueName: \"kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.560249 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.560041 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.560469 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.560450 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.560525 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.560502 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.560562 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.560502 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.578763 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.578734 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9c8w\" (UniqueName: \"kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.578888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.578834 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hbk\" (UniqueName: \"kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.579200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.579178 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbzt\" (UniqueName: \"kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.610674 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.610643 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:00.622680 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.622653 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:00.661929 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.661886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z62k\" (UniqueName: \"kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.662116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.662033 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.662708 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.662683 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.669629 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.669570 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:00.684530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.682271 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:00.700993 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.700351 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z62k\" (UniqueName: \"kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.736324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.736291 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:00.745443 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.745229 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:00.923925 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:00.923883 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85"] Oct 30 14:00:00.926336 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:00:00.926308 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35f61f6_7860_4247_9e3d_b21ad5f66b79.slice/crio-33f11b69e8ade826fc23fdd8267fa78a4f552b74061a188e803c8b3638674144 WatchSource:0}: Error finding container 33f11b69e8ade826fc23fdd8267fa78a4f552b74061a188e803c8b3638674144: Status 404 returned error can't find the container with id 33f11b69e8ade826fc23fdd8267fa78a4f552b74061a188e803c8b3638674144 Oct 30 14:00:01.035436 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.035409 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh"] Oct 30 14:00:01.047522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.047484 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k"] Oct 30 14:00:01.094295 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.094217 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl"] Oct 30 14:00:01.101224 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.101198 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w"] Oct 30 14:00:01.132368 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.132335 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67"] Oct 30 14:00:01.142713 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:00:01.142679 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814dfafd_ce29_4857_829c_56024ac85a80.slice/crio-d120b1af6c9541954e802b58591e4a7a02aef829365be9496d962ddb3a11483b WatchSource:0}: Error finding container d120b1af6c9541954e802b58591e4a7a02aef829365be9496d962ddb3a11483b: Status 404 returned error can't find the container with id d120b1af6c9541954e802b58591e4a7a02aef829365be9496d962ddb3a11483b Oct 30 14:00:01.142978 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:00:01.142955 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9619f1ee_11f4_4005_96bb_8644b95977e0.slice/crio-cbdeb84ebb3219000c4dc3e640258a7325e203dd62cd3d605a731e14d8c80d60 WatchSource:0}: Error finding container cbdeb84ebb3219000c4dc3e640258a7325e203dd62cd3d605a731e14d8c80d60: Status 404 returned error can't find the container with id cbdeb84ebb3219000c4dc3e640258a7325e203dd62cd3d605a731e14d8c80d60 Oct 30 14:00:01.143880 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:00:01.143853 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fecee25_de7d_456b_9b2e_e4907ea32af5.slice/crio-aebeb2f7cda039724109b519aff4f09e158c0e3a09027a238dafb4cbef11ad55 WatchSource:0}: Error finding container aebeb2f7cda039724109b519aff4f09e158c0e3a09027a238dafb4cbef11ad55: Status 404 returned error can't find the container with id aebeb2f7cda039724109b519aff4f09e158c0e3a09027a238dafb4cbef11ad55 Oct 30 14:00:01.863774 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.863728 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerStarted","Data":"826a75f82a7e829878781847cda6b480aa555cc343c25043f2796653e1fd5f02"} Oct 30 14:00:01.864305 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.863799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerStarted","Data":"062a3679b76327d437bcd6c35a559c9e754aadcf5ec943a07aa7e72c8acd1cb8"} Oct 30 14:00:01.865787 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.865752 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerStarted","Data":"802aaa10b5a8639be6583a9b8d3b97c6c54c758c95c99818a47536eb3a3ac5eb"} Oct 30 14:00:01.865927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.865813 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerStarted","Data":"d120b1af6c9541954e802b58591e4a7a02aef829365be9496d962ddb3a11483b"} Oct 30 14:00:01.867978 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.867949 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerStarted","Data":"5d2301cea3fa1f0ab0cf1c9644833ea0b442fbe2e6c7f877e2d24d38a16e9e42"} Oct 30 14:00:01.868126 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.868004 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerStarted","Data":"5465777fcae9606333a04c270a082c3a282c7e6ab371f3531d550ce19811f2a6"} Oct 30 14:00:01.869646 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.869619 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerStarted","Data":"4e75479a251377f652ba0660327d9679e1981ad814bce223d35aeb7efb1e0f0c"} Oct 30 14:00:01.869769 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.869665 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerStarted","Data":"33f11b69e8ade826fc23fdd8267fa78a4f552b74061a188e803c8b3638674144"} Oct 30 14:00:01.871349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.871321 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerStarted","Data":"1855a58bb89db2b6de22f377eb1ea85a9c772ebfea0e3d4340b712b3c709c638"} Oct 30 14:00:01.871484 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.871367 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerStarted","Data":"aebeb2f7cda039724109b519aff4f09e158c0e3a09027a238dafb4cbef11ad55"} Oct 30 14:00:01.872905 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.872881 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerStarted","Data":"92869cb5550189660a7d21444e7265ec544103db1b123e2929972c42e9e1f409"} Oct 30 14:00:01.872994 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.872919 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerStarted","Data":"cbdeb84ebb3219000c4dc3e640258a7325e203dd62cd3d605a731e14d8c80d60"} Oct 30 14:00:01.918107 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.918037 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" podStartSLOduration=1.918017329 podStartE2EDuration="1.918017329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:01.914923213 +0000 UTC m=+3377.781541385" watchObservedRunningTime="2026-10-30 14:00:01.918017329 +0000 UTC m=+3377.784635505" Oct 30 14:00:01.963512 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:01.963450 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" podStartSLOduration=1.963435491 podStartE2EDuration="1.963435491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:01.963171607 +0000 UTC m=+3377.829789780" watchObservedRunningTime="2026-10-30 14:00:01.963435491 +0000 UTC m=+3377.830053664" Oct 30 14:00:02.009257 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.009202 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" podStartSLOduration=2.009184882 podStartE2EDuration="2.009184882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.008338492 +0000 UTC m=+3377.874956666" watchObservedRunningTime="2026-10-30 14:00:02.009184882 +0000 UTC m=+3377.875803058" Oct 30 14:00:02.051707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.051650 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" podStartSLOduration=2.051632427 podStartE2EDuration="2.051632427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.050306454 +0000 UTC m=+3377.916924628" watchObservedRunningTime="2026-10-30 14:00:02.051632427 +0000 UTC m=+3377.918250602" Oct 30 14:00:02.091309 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.091200 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" podStartSLOduration=2.091185948 podStartE2EDuration="2.091185948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.087822737 +0000 UTC m=+3377.954440910" watchObservedRunningTime="2026-10-30 14:00:02.091185948 +0000 UTC m=+3377.957804121" Oct 30 14:00:02.153631 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.153077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:02.178817 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.178780 5693 patch_prober.go:29] interesting pod/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:00:02.179118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.179081 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:00:02.886497 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.886399 5693 generic.go:356] "Generic (PLEG): container finished" podID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerID="1855a58bb89db2b6de22f377eb1ea85a9c772ebfea0e3d4340b712b3c709c638" exitCode=60 Oct 30 14:00:02.886497 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.886478 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerDied","Data":"1855a58bb89db2b6de22f377eb1ea85a9c772ebfea0e3d4340b712b3c709c638"} Oct 30 14:00:02.887067 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.886978 5693 scope.go:122] "RemoveContainer" containerID="1855a58bb89db2b6de22f377eb1ea85a9c772ebfea0e3d4340b712b3c709c638" Oct 30 14:00:02.889278 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.888787 5693 generic.go:356] "Generic (PLEG): container finished" podID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerID="92869cb5550189660a7d21444e7265ec544103db1b123e2929972c42e9e1f409" exitCode=60 Oct 30 14:00:02.889278 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.888885 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerDied","Data":"92869cb5550189660a7d21444e7265ec544103db1b123e2929972c42e9e1f409"} Oct 30 14:00:02.889278 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.889212 5693 scope.go:122] "RemoveContainer" containerID="92869cb5550189660a7d21444e7265ec544103db1b123e2929972c42e9e1f409" Oct 30 14:00:02.907697 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.907654 5693 generic.go:356] "Generic (PLEG): container finished" podID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerID="826a75f82a7e829878781847cda6b480aa555cc343c25043f2796653e1fd5f02" exitCode=60 Oct 30 14:00:02.907857 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.907804 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerDied","Data":"826a75f82a7e829878781847cda6b480aa555cc343c25043f2796653e1fd5f02"} Oct 30 14:00:02.908252 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.908224 5693 scope.go:122] "RemoveContainer" containerID="826a75f82a7e829878781847cda6b480aa555cc343c25043f2796653e1fd5f02" Oct 30 14:00:02.910630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.910529 5693 generic.go:356] "Generic (PLEG): container finished" podID="814dfafd-ce29-4857-829c-56024ac85a80" containerID="802aaa10b5a8639be6583a9b8d3b97c6c54c758c95c99818a47536eb3a3ac5eb" exitCode=60 Oct 30 14:00:02.910744 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.910649 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerDied","Data":"802aaa10b5a8639be6583a9b8d3b97c6c54c758c95c99818a47536eb3a3ac5eb"} Oct 30 14:00:02.911117 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.911024 5693 scope.go:122] "RemoveContainer" containerID="802aaa10b5a8639be6583a9b8d3b97c6c54c758c95c99818a47536eb3a3ac5eb" Oct 30 14:00:02.931972 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.931401 5693 generic.go:356] "Generic (PLEG): container finished" podID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerID="5d2301cea3fa1f0ab0cf1c9644833ea0b442fbe2e6c7f877e2d24d38a16e9e42" exitCode=60 Oct 30 14:00:02.931972 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.931503 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerDied","Data":"5d2301cea3fa1f0ab0cf1c9644833ea0b442fbe2e6c7f877e2d24d38a16e9e42"} Oct 30 14:00:02.933391 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.932045 5693 scope.go:122] "RemoveContainer" containerID="5d2301cea3fa1f0ab0cf1c9644833ea0b442fbe2e6c7f877e2d24d38a16e9e42" Oct 30 14:00:02.942928 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.942866 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" podStartSLOduration=2.94284696 podStartE2EDuration="2.94284696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.138765395 +0000 UTC m=+3378.005383584" watchObservedRunningTime="2026-10-30 14:00:02.94284696 +0000 UTC m=+3378.809465136" Oct 30 14:00:02.967289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.967185 5693 generic.go:356] "Generic (PLEG): container finished" podID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerID="d582f0b0ac880640d6716c4789455b8e25fe10e2801ab91faa39534e530b98e4" exitCode=0 Oct 30 14:00:02.967498 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.967376 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" event={"ID":"1836005b-46a2-45c3-b269-75bfbf3cabfb","Type":"ContainerDied","Data":"d582f0b0ac880640d6716c4789455b8e25fe10e2801ab91faa39534e530b98e4"} Oct 30 14:00:02.967498 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.967455 5693 scope.go:122] "RemoveContainer" containerID="15281020e36f866306a87bac0555bc0effee7dceeb9c4d83b7b55de42fc402d6" Oct 30 14:00:02.983172 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.983141 5693 generic.go:356] "Generic (PLEG): container finished" podID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerID="4e75479a251377f652ba0660327d9679e1981ad814bce223d35aeb7efb1e0f0c" exitCode=60 Oct 30 14:00:02.984467 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.983895 5693 scope.go:122] "RemoveContainer" containerID="4e75479a251377f652ba0660327d9679e1981ad814bce223d35aeb7efb1e0f0c" Oct 30 14:00:02.985108 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:02.983921 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerDied","Data":"4e75479a251377f652ba0660327d9679e1981ad814bce223d35aeb7efb1e0f0c"} Oct 30 14:00:03.015740 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.015714 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 14:00:03.112272 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112235 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.112435 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112363 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.112435 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112426 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.112523 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112506 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.112651 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112592 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.112861 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112767 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:03.112927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112812 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:03.112927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112895 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.112927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112916 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:03.113046 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.112965 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.113111 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.113065 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.113111 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.113081 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:03.113230 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.113167 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo\") pod \"1836005b-46a2-45c3-b269-75bfbf3cabfb\" (UID: \"1836005b-46a2-45c3-b269-75bfbf3cabfb\") " Oct 30 14:00:03.113369 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.113336 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:03.113961 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.113937 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.114104 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.114088 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.114201 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.114188 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1836005b-46a2-45c3-b269-75bfbf3cabfb-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.114289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.114278 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.114436 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.114370 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.132302 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.132260 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:03.132302 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.132280 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "kube-api-access-fsvd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:03.132503 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.132386 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:03.132754 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.132732 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo" pod "1836005b-46a2-45c3-b269-75bfbf3cabfb" (UID: "1836005b-46a2-45c3-b269-75bfbf3cabfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:00:03.215771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.215736 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsvd6\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-kube-api-access-fsvd6\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.215771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.215765 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.215771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.215776 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.215771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.215784 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1836005b-46a2-45c3-b269-75bfbf3cabfb-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:03.939781 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.939696 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:03.996716 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.996614 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerStarted","Data":"f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06"} Oct 30 14:00:03.998724 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:03.998694 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerStarted","Data":"e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8"} Oct 30 14:00:04.000543 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.000512 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerStarted","Data":"d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842"} Oct 30 14:00:04.002173 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.002146 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" event={"ID":"1836005b-46a2-45c3-b269-75bfbf3cabfb","Type":"ContainerDied","Data":"d1472d705d8310c696f2ca97b5ed6db708236c67074e1a5bab62c53b7bbedee8"} Oct 30 14:00:04.002291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.002193 5693 scope.go:122] "RemoveContainer" containerID="d582f0b0ac880640d6716c4789455b8e25fe10e2801ab91faa39534e530b98e4" Oct 30 14:00:04.002291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.002264 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs" Oct 30 14:00:04.007414 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.007382 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerStarted","Data":"4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4"} Oct 30 14:00:04.009841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.009816 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerStarted","Data":"039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e"} Oct 30 14:00:04.012084 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.012060 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerStarted","Data":"e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1"} Oct 30 14:00:04.213568 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.211164 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs"] Oct 30 14:00:04.218665 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.218628 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-996b8b-openshift-default-868b7bd757-bl5qs"] Oct 30 14:00:04.260666 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:04.260550 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" path="/var/lib/kubelet/pods/1836005b-46a2-45c3-b269-75bfbf3cabfb/volumes" Oct 30 14:00:05.022667 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.022587 5693 generic.go:356] "Generic (PLEG): container finished" podID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerID="039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e" exitCode=60 Oct 30 14:00:05.023182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.022677 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerDied","Data":"039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e"} Oct 30 14:00:05.023182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.022766 5693 scope.go:122] "RemoveContainer" containerID="1855a58bb89db2b6de22f377eb1ea85a9c772ebfea0e3d4340b712b3c709c638" Oct 30 14:00:05.023182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.023120 5693 scope.go:122] "RemoveContainer" containerID="039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e" Oct 30 14:00:05.023554 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:05.023531 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl_odh-ai-gateway-infra(9fecee25-de7d-456b-9b2e-e4907ea32af5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" Oct 30 14:00:05.024982 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.024951 5693 generic.go:356] "Generic (PLEG): container finished" podID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerID="e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1" exitCode=60 Oct 30 14:00:05.025077 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.025036 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerDied","Data":"e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1"} Oct 30 14:00:05.025347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.025330 5693 scope.go:122] "RemoveContainer" containerID="e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1" Oct 30 14:00:05.025712 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:05.025687 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w_odh-ai-gateway-infra(9619f1ee-11f4-4005-96bb-8644b95977e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" Oct 30 14:00:05.026869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.026849 5693 generic.go:356] "Generic (PLEG): container finished" podID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerID="f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06" exitCode=60 Oct 30 14:00:05.026956 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.026936 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerDied","Data":"f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06"} Oct 30 14:00:05.027203 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.027186 5693 scope.go:122] "RemoveContainer" containerID="f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06" Oct 30 14:00:05.027552 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:05.027523 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k_odh-ai-gateway-infra(d9408ace-b372-4dde-8af1-f6f599bc9feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" Oct 30 14:00:05.028882 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.028861 5693 generic.go:356] "Generic (PLEG): container finished" podID="814dfafd-ce29-4857-829c-56024ac85a80" containerID="e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8" exitCode=60 Oct 30 14:00:05.028989 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.028939 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerDied","Data":"e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8"} Oct 30 14:00:05.029186 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.029168 5693 scope.go:122] "RemoveContainer" containerID="e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8" Oct 30 14:00:05.029471 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:05.029451 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67_odh-ai-gateway-infra(814dfafd-ce29-4857-829c-56024ac85a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" podUID="814dfafd-ce29-4857-829c-56024ac85a80" Oct 30 14:00:05.030880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.030857 5693 generic.go:356] "Generic (PLEG): container finished" podID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerID="d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842" exitCode=60 Oct 30 14:00:05.031182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.031157 5693 scope.go:122] "RemoveContainer" containerID="d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842" Oct 30 14:00:05.031253 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.031210 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerDied","Data":"d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842"} Oct 30 14:00:05.031490 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:05.031469 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh_odh-ai-gateway-infra(f80827f5-4b51-42c8-bf30-75ff4c262fd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" Oct 30 14:00:05.033738 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.033715 5693 generic.go:356] "Generic (PLEG): container finished" podID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerID="4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4" exitCode=60 Oct 30 14:00:05.033840 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.033788 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerDied","Data":"4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4"} Oct 30 14:00:05.034019 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.034002 5693 scope.go:122] "RemoveContainer" containerID="4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4" Oct 30 14:00:05.034279 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:05.034260 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85_odh-ai-gateway-infra(b35f61f6-7860-4247-9e3d-b21ad5f66b79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" Oct 30 14:00:05.037663 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.037646 5693 scope.go:122] "RemoveContainer" containerID="92869cb5550189660a7d21444e7265ec544103db1b123e2929972c42e9e1f409" Oct 30 14:00:05.052227 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.052205 5693 scope.go:122] "RemoveContainer" containerID="826a75f82a7e829878781847cda6b480aa555cc343c25043f2796653e1fd5f02" Oct 30 14:00:05.075328 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.075277 5693 scope.go:122] "RemoveContainer" containerID="802aaa10b5a8639be6583a9b8d3b97c6c54c758c95c99818a47536eb3a3ac5eb" Oct 30 14:00:05.091373 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.091346 5693 scope.go:122] "RemoveContainer" containerID="5d2301cea3fa1f0ab0cf1c9644833ea0b442fbe2e6c7f877e2d24d38a16e9e42" Oct 30 14:00:05.110008 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.109970 5693 scope.go:122] "RemoveContainer" containerID="4e75479a251377f652ba0660327d9679e1981ad814bce223d35aeb7efb1e0f0c" Oct 30 14:00:05.644717 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:05.637542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:06.044226 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:06.044196 5693 scope.go:122] "RemoveContainer" containerID="e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8" Oct 30 14:00:06.044699 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:06.044580 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67_odh-ai-gateway-infra(814dfafd-ce29-4857-829c-56024ac85a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" podUID="814dfafd-ce29-4857-829c-56024ac85a80" Oct 30 14:00:06.045887 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:06.045862 5693 scope.go:122] "RemoveContainer" containerID="d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842" Oct 30 14:00:06.046222 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:06.046201 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh_odh-ai-gateway-infra(f80827f5-4b51-42c8-bf30-75ff4c262fd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" Oct 30 14:00:06.047480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:06.047454 5693 scope.go:122] "RemoveContainer" containerID="4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4" Oct 30 14:00:06.047758 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:06.047733 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85_odh-ai-gateway-infra(b35f61f6-7860-4247-9e3d-b21ad5f66b79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" Oct 30 14:00:06.049146 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:06.049126 5693 scope.go:122] "RemoveContainer" containerID="039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e" Oct 30 14:00:06.049420 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:06.049390 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl_odh-ai-gateway-infra(9fecee25-de7d-456b-9b2e-e4907ea32af5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" Oct 30 14:00:06.050668 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:06.050651 5693 scope.go:122] "RemoveContainer" containerID="e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1" Oct 30 14:00:06.050910 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:06.050891 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w_odh-ai-gateway-infra(9619f1ee-11f4-4005-96bb-8644b95977e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" Oct 30 14:00:06.052327 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:06.052300 5693 scope.go:122] "RemoveContainer" containerID="f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06" Oct 30 14:00:06.052563 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:06.052543 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k_odh-ai-gateway-infra(d9408ace-b372-4dde-8af1-f6f599bc9feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" Oct 30 14:00:07.433136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:07.433095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:09.323628 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.320700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:09.494317 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.494280 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4"] Oct 30 14:00:09.495176 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.495153 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" Oct 30 14:00:09.495176 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.495174 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" Oct 30 14:00:09.495385 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.495206 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" Oct 30 14:00:09.495385 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.495212 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" Oct 30 14:00:09.495385 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.495339 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" Oct 30 14:00:09.500860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.500836 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.507057 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.507033 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:09.516011 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.515988 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:00:09.532457 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.532426 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4"] Oct 30 14:00:09.541286 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.541254 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.541567 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.541546 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.541652 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.541615 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtnd\" (UniqueName: \"kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.642895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.642801 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.642895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.642849 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtnd\" (UniqueName: \"kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.643139 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.642951 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.643555 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.643530 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.645297 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.645271 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.668088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.668060 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtnd\" (UniqueName: \"kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd\") pod \"maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:09.816346 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:09.816303 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:10.134564 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:10.134541 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4"] Oct 30 14:00:10.136546 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:00:10.136515 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91243fe1_ca46_4a14_97f2_821242a8edf9.slice/crio-cee7d28d6527c176126e92b76b775a1dc114daee980dead2d0c7c3682adb7292 WatchSource:0}: Error finding container cee7d28d6527c176126e92b76b775a1dc114daee980dead2d0c7c3682adb7292: Status 404 returned error can't find the container with id cee7d28d6527c176126e92b76b775a1dc114daee980dead2d0c7c3682adb7292 Oct 30 14:00:11.093125 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:11.093080 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" event={"ID":"91243fe1-ca46-4a14-97f2-821242a8edf9","Type":"ContainerStarted","Data":"8107aef006e2b078042a9695826ac97f94755c21dce3126a2dc20dac71267d56"} Oct 30 14:00:11.093635 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:11.093134 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" event={"ID":"91243fe1-ca46-4a14-97f2-821242a8edf9","Type":"ContainerStarted","Data":"cee7d28d6527c176126e92b76b775a1dc114daee980dead2d0c7c3682adb7292"} Oct 30 14:00:11.140383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:11.140326 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" podStartSLOduration=2.140310934 podStartE2EDuration="2.140310934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:11.135148252 +0000 UTC m=+3387.001766424" watchObservedRunningTime="2026-10-30 14:00:11.140310934 +0000 UTC m=+3387.006929106" Oct 30 14:00:11.500360 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:11.500322 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:12.103970 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:12.103937 5693 generic.go:356] "Generic (PLEG): container finished" podID="91243fe1-ca46-4a14-97f2-821242a8edf9" containerID="8107aef006e2b078042a9695826ac97f94755c21dce3126a2dc20dac71267d56" exitCode=0 Oct 30 14:00:12.104406 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:12.104030 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" event={"ID":"91243fe1-ca46-4a14-97f2-821242a8edf9","Type":"ContainerDied","Data":"8107aef006e2b078042a9695826ac97f94755c21dce3126a2dc20dac71267d56"} Oct 30 14:00:12.882725 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:12.882672 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297"] Oct 30 14:00:12.883048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:12.883019 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" podUID="26296c50-cd6e-4627-891d-705cf55c1b51" containerName="maas-api" containerID="cri-o://7116b796b607373d1a0e7c1d770b2b8f02a349423eb98bca2fb3ea0b0e63e88a" gracePeriod=30 Oct 30 14:00:13.016048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.016017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:13.115472 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.115428 5693 generic.go:356] "Generic (PLEG): container finished" podID="26296c50-cd6e-4627-891d-705cf55c1b51" containerID="7116b796b607373d1a0e7c1d770b2b8f02a349423eb98bca2fb3ea0b0e63e88a" exitCode=0 Oct 30 14:00:13.115968 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.115497 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" event={"ID":"26296c50-cd6e-4627-891d-705cf55c1b51","Type":"ContainerDied","Data":"7116b796b607373d1a0e7c1d770b2b8f02a349423eb98bca2fb3ea0b0e63e88a"} Oct 30 14:00:13.275035 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.274999 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 14:00:13.398658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.380983 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs\") pod \"26296c50-cd6e-4627-891d-705cf55c1b51\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " Oct 30 14:00:13.398658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.381139 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls\") pod \"26296c50-cd6e-4627-891d-705cf55c1b51\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " Oct 30 14:00:13.398658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.381474 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87\" (UniqueName: \"kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87\") pod \"26296c50-cd6e-4627-891d-705cf55c1b51\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " Oct 30 14:00:13.398658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.381562 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca\") pod \"26296c50-cd6e-4627-891d-705cf55c1b51\" (UID: \"26296c50-cd6e-4627-891d-705cf55c1b51\") " Oct 30 14:00:13.398658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.383431 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca" pod "26296c50-cd6e-4627-891d-705cf55c1b51" (UID: "26296c50-cd6e-4627-891d-705cf55c1b51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:13.456157 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.432707 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87" pod "26296c50-cd6e-4627-891d-705cf55c1b51" (UID: "26296c50-cd6e-4627-891d-705cf55c1b51"). InnerVolumeSpecName "kube-api-access-hbc87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:13.456157 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.434302 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs" pod "26296c50-cd6e-4627-891d-705cf55c1b51" (UID: "26296c50-cd6e-4627-891d-705cf55c1b51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:00:13.456157 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.434406 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls" pod "26296c50-cd6e-4627-891d-705cf55c1b51" (UID: "26296c50-cd6e-4627-891d-705cf55c1b51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:00:13.507765 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.497724 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:13.517675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.516751 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbc87\" (UniqueName: \"kubernetes.io/projected/26296c50-cd6e-4627-891d-705cf55c1b51-kube-api-access-hbc87\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.517675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.516785 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26296c50-cd6e-4627-891d-705cf55c1b51-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.517675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.516801 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.517675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.516812 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26296c50-cd6e-4627-891d-705cf55c1b51-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.617673 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.617634 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca\") pod \"91243fe1-ca46-4a14-97f2-821242a8edf9\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " Oct 30 14:00:13.617877 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.617744 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd\" (UniqueName: \"kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd\") pod \"91243fe1-ca46-4a14-97f2-821242a8edf9\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " Oct 30 14:00:13.617877 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.617811 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca\") pod \"91243fe1-ca46-4a14-97f2-821242a8edf9\" (UID: \"91243fe1-ca46-4a14-97f2-821242a8edf9\") " Oct 30 14:00:13.618082 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.618049 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca" pod "91243fe1-ca46-4a14-97f2-821242a8edf9" (UID: "91243fe1-ca46-4a14-97f2-821242a8edf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:13.618493 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.618471 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-openshift-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.620113 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.620085 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca" pod "91243fe1-ca46-4a14-97f2-821242a8edf9" (UID: "91243fe1-ca46-4a14-97f2-821242a8edf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:00:13.620216 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.620133 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd" pod "91243fe1-ca46-4a14-97f2-821242a8edf9" (UID: "91243fe1-ca46-4a14-97f2-821242a8edf9"). InnerVolumeSpecName "kube-api-access-hqtnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:13.720356 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.720277 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqtnd\" (UniqueName: \"kubernetes.io/projected/91243fe1-ca46-4a14-97f2-821242a8edf9-kube-api-access-hqtnd\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.720356 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.720306 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91243fe1-ca46-4a14-97f2-821242a8edf9-maas-api-xks-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:13.830874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:13.830829 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:14.095923 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.095892 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:14.125488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.125451 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" Oct 30 14:00:14.125488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.125477 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4" event={"ID":"91243fe1-ca46-4a14-97f2-821242a8edf9","Type":"ContainerDied","Data":"cee7d28d6527c176126e92b76b775a1dc114daee980dead2d0c7c3682adb7292"} Oct 30 14:00:14.125980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.125547 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee7d28d6527c176126e92b76b775a1dc114daee980dead2d0c7c3682adb7292" Oct 30 14:00:14.127183 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.127160 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" event={"ID":"26296c50-cd6e-4627-891d-705cf55c1b51","Type":"ContainerDied","Data":"673ff5f10a27bc8e1f50c16dbfec914a0a46a56586c2ae0ec7449b901c9ee8ef"} Oct 30 14:00:14.127335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.127196 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297" Oct 30 14:00:14.127335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.127214 5693 scope.go:122] "RemoveContainer" containerID="7116b796b607373d1a0e7c1d770b2b8f02a349423eb98bca2fb3ea0b0e63e88a" Oct 30 14:00:14.191039 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.191006 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297"] Oct 30 14:00:14.202808 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.202775 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88352d-bb77558c9-mv297"] Oct 30 14:00:14.256091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.256049 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26296c50-cd6e-4627-891d-705cf55c1b51" path="/var/lib/kubelet/pods/26296c50-cd6e-4627-891d-705cf55c1b51/volumes" Oct 30 14:00:14.463743 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:14.463563 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:16.139797 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:16.139739 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:16.247682 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:16.247650 5693 scope.go:122] "RemoveContainer" containerID="e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8" Oct 30 14:00:17.161027 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:17.160994 5693 generic.go:356] "Generic (PLEG): container finished" podID="814dfafd-ce29-4857-829c-56024ac85a80" containerID="28aa06243f4fb6f61ddf8174ffc418e42e88e3460074ecef932572cae88696e3" exitCode=60 Oct 30 14:00:17.161408 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:17.161067 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerDied","Data":"28aa06243f4fb6f61ddf8174ffc418e42e88e3460074ecef932572cae88696e3"} Oct 30 14:00:17.161408 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:17.161153 5693 scope.go:122] "RemoveContainer" containerID="e08c7377306b5a3aacc550d4867769bf7e037b816dc7484682a6538ecfff24f8" Oct 30 14:00:17.161592 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:17.161576 5693 scope.go:122] "RemoveContainer" containerID="28aa06243f4fb6f61ddf8174ffc418e42e88e3460074ecef932572cae88696e3" Oct 30 14:00:17.162007 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:17.161986 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67_odh-ai-gateway-infra(814dfafd-ce29-4857-829c-56024ac85a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" podUID="814dfafd-ce29-4857-829c-56024ac85a80" Oct 30 14:00:17.247784 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:17.247754 5693 scope.go:122] "RemoveContainer" containerID="f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06" Oct 30 14:00:17.247963 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:17.247930 5693 scope.go:122] "RemoveContainer" containerID="039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e" Oct 30 14:00:18.105142 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.105052 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:18.170762 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.170727 5693 generic.go:356] "Generic (PLEG): container finished" podID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerID="dfe1397efdac2117e37a2be1c9e28c3121040fc446b408c96bd4bbe5654c2770" exitCode=60 Oct 30 14:00:18.171244 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.170772 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerDied","Data":"dfe1397efdac2117e37a2be1c9e28c3121040fc446b408c96bd4bbe5654c2770"} Oct 30 14:00:18.171244 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.170863 5693 scope.go:122] "RemoveContainer" containerID="039fd4f80cb1b5dc7745a864e40ab889ac2d86a7494a87a92f5929c0e2c7fd6e" Oct 30 14:00:18.171420 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.171393 5693 scope.go:122] "RemoveContainer" containerID="dfe1397efdac2117e37a2be1c9e28c3121040fc446b408c96bd4bbe5654c2770" Oct 30 14:00:18.171840 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:18.171816 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl_odh-ai-gateway-infra(9fecee25-de7d-456b-9b2e-e4907ea32af5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" Oct 30 14:00:18.173178 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.173152 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerStarted","Data":"c7609e88550c6b07d607a7c6ef3908d92a3d3558ecddca57bd46b81229c4bcc3"} Oct 30 14:00:18.241368 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.241335 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67"] Oct 30 14:00:18.259630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.259580 5693 scope.go:122] "RemoveContainer" containerID="4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4" Oct 30 14:00:18.260200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.260161 5693 scope.go:122] "RemoveContainer" containerID="e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1" Oct 30 14:00:18.260906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.260700 5693 scope.go:122] "RemoveContainer" containerID="d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842" Oct 30 14:00:18.618830 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.618799 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:18.732165 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.732136 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca\") pod \"814dfafd-ce29-4857-829c-56024ac85a80\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " Oct 30 14:00:18.732558 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.732531 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca" pod "814dfafd-ce29-4857-829c-56024ac85a80" (UID: "814dfafd-ce29-4857-829c-56024ac85a80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:18.732693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.732569 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp\" (UniqueName: \"kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp\") pod \"814dfafd-ce29-4857-829c-56024ac85a80\" (UID: \"814dfafd-ce29-4857-829c-56024ac85a80\") " Oct 30 14:00:18.733084 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.733065 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/814dfafd-ce29-4857-829c-56024ac85a80-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:18.828279 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.828222 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp" pod "814dfafd-ce29-4857-829c-56024ac85a80" (UID: "814dfafd-ce29-4857-829c-56024ac85a80"). InnerVolumeSpecName "kube-api-access-s5ktp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:18.834921 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:18.834888 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ktp\" (UniqueName: \"kubernetes.io/projected/814dfafd-ce29-4857-829c-56024ac85a80-kube-api-access-s5ktp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:19.186798 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.186697 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerStarted","Data":"4be2e6e1177edb2ee7946170d290824839387fc8ceb307ed4bdeefe732ef5bbe"} Oct 30 14:00:19.188743 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.188705 5693 generic.go:356] "Generic (PLEG): container finished" podID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerID="c7609e88550c6b07d607a7c6ef3908d92a3d3558ecddca57bd46b81229c4bcc3" exitCode=60 Oct 30 14:00:19.188858 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.188834 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerDied","Data":"c7609e88550c6b07d607a7c6ef3908d92a3d3558ecddca57bd46b81229c4bcc3"} Oct 30 14:00:19.188921 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.188897 5693 scope.go:122] "RemoveContainer" containerID="f38f5909dbaed4ed6a9c2e19f4d58d8345d83faa833f126d79cc24b745d34e06" Oct 30 14:00:19.189366 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.189341 5693 scope.go:122] "RemoveContainer" containerID="c7609e88550c6b07d607a7c6ef3908d92a3d3558ecddca57bd46b81229c4bcc3" Oct 30 14:00:19.189774 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:19.189750 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k_odh-ai-gateway-infra(d9408ace-b372-4dde-8af1-f6f599bc9feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" Oct 30 14:00:19.191306 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.191288 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" Oct 30 14:00:19.191419 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.191316 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67" event={"ID":"814dfafd-ce29-4857-829c-56024ac85a80","Type":"ContainerDied","Data":"d120b1af6c9541954e802b58591e4a7a02aef829365be9496d962ddb3a11483b"} Oct 30 14:00:19.193671 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.193639 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerStarted","Data":"26dd8c60fa44571012fa18115d09fdacab6a47e902214563149b55eeb3b53ec3"} Oct 30 14:00:19.195905 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.195882 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerStarted","Data":"d4975e62d0d0b21df68d831e3afa747bc607809d59c00defdd35f1b2a3e0516d"} Oct 30 14:00:19.204967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.204943 5693 scope.go:122] "RemoveContainer" containerID="28aa06243f4fb6f61ddf8174ffc418e42e88e3460074ecef932572cae88696e3" Oct 30 14:00:19.237149 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.237109 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl"] Oct 30 14:00:19.283904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.283836 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k"] Oct 30 14:00:19.398394 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.398362 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67"] Oct 30 14:00:19.403045 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.403019 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889480-lqs67"] Oct 30 14:00:19.424889 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.424862 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:19.558237 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.558209 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca\") pod \"9fecee25-de7d-456b-9b2e-e4907ea32af5\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " Oct 30 14:00:19.558375 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.558286 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk\" (UniqueName: \"kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk\") pod \"9fecee25-de7d-456b-9b2e-e4907ea32af5\" (UID: \"9fecee25-de7d-456b-9b2e-e4907ea32af5\") " Oct 30 14:00:19.558578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.558553 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca" pod "9fecee25-de7d-456b-9b2e-e4907ea32af5" (UID: "9fecee25-de7d-456b-9b2e-e4907ea32af5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:19.558957 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.558939 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fecee25-de7d-456b-9b2e-e4907ea32af5-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:19.560549 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.560522 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk" pod "9fecee25-de7d-456b-9b2e-e4907ea32af5" (UID: "9fecee25-de7d-456b-9b2e-e4907ea32af5"). InnerVolumeSpecName "kube-api-access-b7hbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:19.660736 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:19.660704 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hbk\" (UniqueName: \"kubernetes.io/projected/9fecee25-de7d-456b-9b2e-e4907ea32af5-kube-api-access-b7hbk\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:20.044147 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.044020 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:20.225062 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.225022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:20.248430 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.248382 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" Oct 30 14:00:20.251513 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.251484 5693 generic.go:356] "Generic (PLEG): container finished" podID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerID="4be2e6e1177edb2ee7946170d290824839387fc8ceb307ed4bdeefe732ef5bbe" exitCode=60 Oct 30 14:00:20.259483 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.259324 5693 generic.go:356] "Generic (PLEG): container finished" podID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerID="26dd8c60fa44571012fa18115d09fdacab6a47e902214563149b55eeb3b53ec3" exitCode=60 Oct 30 14:00:20.261390 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.261367 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814dfafd-ce29-4857-829c-56024ac85a80" path="/var/lib/kubelet/pods/814dfafd-ce29-4857-829c-56024ac85a80/volumes" Oct 30 14:00:20.262030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262009 5693 generic.go:356] "Generic (PLEG): container finished" podID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerID="d4975e62d0d0b21df68d831e3afa747bc607809d59c00defdd35f1b2a3e0516d" exitCode=60 Oct 30 14:00:20.262124 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262082 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl" event={"ID":"9fecee25-de7d-456b-9b2e-e4907ea32af5","Type":"ContainerDied","Data":"aebeb2f7cda039724109b519aff4f09e158c0e3a09027a238dafb4cbef11ad55"} Oct 30 14:00:20.262124 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerDied","Data":"4be2e6e1177edb2ee7946170d290824839387fc8ceb307ed4bdeefe732ef5bbe"} Oct 30 14:00:20.262202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262134 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w"] Oct 30 14:00:20.262202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262147 5693 scope.go:122] "RemoveContainer" containerID="dfe1397efdac2117e37a2be1c9e28c3121040fc446b408c96bd4bbe5654c2770" Oct 30 14:00:20.262202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262163 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerDied","Data":"26dd8c60fa44571012fa18115d09fdacab6a47e902214563149b55eeb3b53ec3"} Oct 30 14:00:20.262202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerDied","Data":"d4975e62d0d0b21df68d831e3afa747bc607809d59c00defdd35f1b2a3e0516d"} Oct 30 14:00:20.262552 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262532 5693 scope.go:122] "RemoveContainer" containerID="d4975e62d0d0b21df68d831e3afa747bc607809d59c00defdd35f1b2a3e0516d" Oct 30 14:00:20.262710 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.262693 5693 scope.go:122] "RemoveContainer" containerID="26dd8c60fa44571012fa18115d09fdacab6a47e902214563149b55eeb3b53ec3" Oct 30 14:00:20.262920 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:20.262824 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85_odh-ai-gateway-infra(b35f61f6-7860-4247-9e3d-b21ad5f66b79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" Oct 30 14:00:20.262995 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:20.262969 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh_odh-ai-gateway-infra(f80827f5-4b51-42c8-bf30-75ff4c262fd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" Oct 30 14:00:20.283898 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.283866 5693 scope.go:122] "RemoveContainer" containerID="e7e2e043b5de0ee3152e9884ebb549c00cd5fbf690dcba6f2319fec985a4d4e1" Oct 30 14:00:20.310952 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.310925 5693 scope.go:122] "RemoveContainer" containerID="d9e8a032e93fc54f110cc247e766037be511051b41a4f45b7e0f9be692903842" Oct 30 14:00:20.324496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.324465 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh"] Oct 30 14:00:20.329133 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.329107 5693 scope.go:122] "RemoveContainer" containerID="4500eea25556b5650f20d7b379295c99ee1b2675214d5c669762557a61c54df4" Oct 30 14:00:20.385638 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.384692 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85"] Oct 30 14:00:20.482770 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.482562 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl"] Oct 30 14:00:20.487639 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.487584 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889480-nltjl"] Oct 30 14:00:20.489675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.489652 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:20.561345 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.561279 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:20.618783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.612262 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt\" (UniqueName: \"kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt\") pod \"d9408ace-b372-4dde-8af1-f6f599bc9feb\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " Oct 30 14:00:20.618783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.612550 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca\") pod \"d9408ace-b372-4dde-8af1-f6f599bc9feb\" (UID: \"d9408ace-b372-4dde-8af1-f6f599bc9feb\") " Oct 30 14:00:20.624093 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.624033 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt" pod "d9408ace-b372-4dde-8af1-f6f599bc9feb" (UID: "d9408ace-b372-4dde-8af1-f6f599bc9feb"). InnerVolumeSpecName "kube-api-access-tmbzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:20.625907 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.624961 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca" pod "d9408ace-b372-4dde-8af1-f6f599bc9feb" (UID: "d9408ace-b372-4dde-8af1-f6f599bc9feb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:20.638621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.634853 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmbzt\" (UniqueName: \"kubernetes.io/projected/d9408ace-b372-4dde-8af1-f6f599bc9feb-kube-api-access-tmbzt\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:20.638621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.634885 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9408ace-b372-4dde-8af1-f6f599bc9feb-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:20.693620 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.693527 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4"] Oct 30 14:00:20.702965 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.693811 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" containerID="cri-o://9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca" gracePeriod=30 Oct 30 14:00:20.735384 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.735355 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca\") pod \"9619f1ee-11f4-4005-96bb-8644b95977e0\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " Oct 30 14:00:20.736014 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.735995 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k\" (UniqueName: \"kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k\") pod \"9619f1ee-11f4-4005-96bb-8644b95977e0\" (UID: \"9619f1ee-11f4-4005-96bb-8644b95977e0\") " Oct 30 14:00:20.737136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.735844 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca" pod "9619f1ee-11f4-4005-96bb-8644b95977e0" (UID: "9619f1ee-11f4-4005-96bb-8644b95977e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:20.742050 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.741668 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k" pod "9619f1ee-11f4-4005-96bb-8644b95977e0" (UID: "9619f1ee-11f4-4005-96bb-8644b95977e0"). InnerVolumeSpecName "kube-api-access-5z62k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:20.838886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.838788 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619f1ee-11f4-4005-96bb-8644b95977e0-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:20.838886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:20.838819 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z62k\" (UniqueName: \"kubernetes.io/projected/9619f1ee-11f4-4005-96bb-8644b95977e0-kube-api-access-5z62k\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:21.275035 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.275002 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" Oct 30 14:00:21.275638 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.275027 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w" event={"ID":"9619f1ee-11f4-4005-96bb-8644b95977e0","Type":"ContainerDied","Data":"cbdeb84ebb3219000c4dc3e640258a7325e203dd62cd3d605a731e14d8c80d60"} Oct 30 14:00:21.275779 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.275702 5693 scope.go:122] "RemoveContainer" containerID="4be2e6e1177edb2ee7946170d290824839387fc8ceb307ed4bdeefe732ef5bbe" Oct 30 14:00:21.276990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.276908 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" Oct 30 14:00:21.276990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.276966 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k" event={"ID":"d9408ace-b372-4dde-8af1-f6f599bc9feb","Type":"ContainerDied","Data":"062a3679b76327d437bcd6c35a559c9e754aadcf5ec943a07aa7e72c8acd1cb8"} Oct 30 14:00:21.290454 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.290312 5693 scope.go:122] "RemoveContainer" containerID="c7609e88550c6b07d607a7c6ef3908d92a3d3558ecddca57bd46b81229c4bcc3" Oct 30 14:00:21.385829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.385644 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k"] Oct 30 14:00:21.396830 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.396798 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889480-2f46k"] Oct 30 14:00:21.440385 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.440351 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w"] Oct 30 14:00:21.448212 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.448185 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889480-mx55w"] Oct 30 14:00:21.508899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.508870 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:21.550820 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.550741 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca\") pod \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " Oct 30 14:00:21.551008 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.550948 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj\" (UniqueName: \"kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj\") pod \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\" (UID: \"b35f61f6-7860-4247-9e3d-b21ad5f66b79\") " Oct 30 14:00:21.552223 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.552193 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca" pod "b35f61f6-7860-4247-9e3d-b21ad5f66b79" (UID: "b35f61f6-7860-4247-9e3d-b21ad5f66b79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:21.553901 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.553878 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj" pod "b35f61f6-7860-4247-9e3d-b21ad5f66b79" (UID: "b35f61f6-7860-4247-9e3d-b21ad5f66b79"). InnerVolumeSpecName "kube-api-access-2wwcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:21.554895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.554876 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:21.652526 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.652485 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w\" (UniqueName: \"kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w\") pod \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " Oct 30 14:00:21.652864 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.652831 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca\") pod \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\" (UID: \"f80827f5-4b51-42c8-bf30-75ff4c262fd7\") " Oct 30 14:00:21.653323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.653280 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca" pod "f80827f5-4b51-42c8-bf30-75ff4c262fd7" (UID: "f80827f5-4b51-42c8-bf30-75ff4c262fd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:21.653483 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.653455 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwcj\" (UniqueName: \"kubernetes.io/projected/b35f61f6-7860-4247-9e3d-b21ad5f66b79-kube-api-access-2wwcj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:21.653483 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.653471 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80827f5-4b51-42c8-bf30-75ff4c262fd7-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:21.653483 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.653481 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f61f6-7860-4247-9e3d-b21ad5f66b79-maas-api-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:21.655270 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.655245 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w" pod "f80827f5-4b51-42c8-bf30-75ff4c262fd7" (UID: "f80827f5-4b51-42c8-bf30-75ff4c262fd7"). InnerVolumeSpecName "kube-api-access-b9c8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:21.755281 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:21.755247 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9c8w\" (UniqueName: \"kubernetes.io/projected/f80827f5-4b51-42c8-bf30-75ff4c262fd7-kube-api-access-b9c8w\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:22.034531 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.026712 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:22.256778 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.256749 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" path="/var/lib/kubelet/pods/9619f1ee-11f4-4005-96bb-8644b95977e0/volumes" Oct 30 14:00:22.257258 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.257243 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" path="/var/lib/kubelet/pods/9fecee25-de7d-456b-9b2e-e4907ea32af5/volumes" Oct 30 14:00:22.257710 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.257696 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" path="/var/lib/kubelet/pods/d9408ace-b372-4dde-8af1-f6f599bc9feb/volumes" Oct 30 14:00:22.305180 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.305149 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" Oct 30 14:00:22.305180 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.305168 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh" event={"ID":"f80827f5-4b51-42c8-bf30-75ff4c262fd7","Type":"ContainerDied","Data":"5465777fcae9606333a04c270a082c3a282c7e6ab371f3531d550ce19811f2a6"} Oct 30 14:00:22.305729 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.305243 5693 scope.go:122] "RemoveContainer" containerID="26dd8c60fa44571012fa18115d09fdacab6a47e902214563149b55eeb3b53ec3" Oct 30 14:00:22.308773 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.308741 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" Oct 30 14:00:22.308882 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.308799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85" event={"ID":"b35f61f6-7860-4247-9e3d-b21ad5f66b79","Type":"ContainerDied","Data":"33f11b69e8ade826fc23fdd8267fa78a4f552b74061a188e803c8b3638674144"} Oct 30 14:00:22.320020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.319994 5693 scope.go:122] "RemoveContainer" containerID="d4975e62d0d0b21df68d831e3afa747bc607809d59c00defdd35f1b2a3e0516d" Oct 30 14:00:22.362755 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.362724 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh"] Oct 30 14:00:22.385773 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.385743 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889480-nj7mh"] Oct 30 14:00:22.422958 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.422926 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85"] Oct 30 14:00:22.426255 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:22.426232 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889480-2sv85"] Oct 30 14:00:23.062644 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:23.062589 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:00:23.830728 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:23.830687 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:24.031026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:24.030986 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:24.184030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:24.183872 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:24.259682 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:24.259645 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" path="/var/lib/kubelet/pods/b35f61f6-7860-4247-9e3d-b21ad5f66b79/volumes" Oct 30 14:00:24.260387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:24.260360 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" path="/var/lib/kubelet/pods/f80827f5-4b51-42c8-bf30-75ff4c262fd7/volumes" Oct 30 14:00:24.364292 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:24.364257 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:25.285015 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:25.284971 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:25.442303 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:25.442265 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:25.777438 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:25.777078 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:25.960644 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:25.960562 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:26.012054 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.011985 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 14:00:26.027434 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027402 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.027634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027471 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.027634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027509 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.027634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027577 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.027634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027623 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027695 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027791 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027693 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027829 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027858 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket\") pod \"b8b856a0-3adb-4403-a497-91b12ad5d85d\" (UID: \"b8b856a0-3adb-4403-a497-91b12ad5d85d\") " Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.027940 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028063 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028264 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:26.028412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028410 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:26.028934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028498 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.028934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028509 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b856a0-3adb-4403-a497-91b12ad5d85d-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.028934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028518 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.028934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028526 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.028934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.028535 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.030120 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.030089 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "kube-api-access-665kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:26.030256 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.030154 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:00:26.030413 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.030384 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:00:26.030535 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.030434 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token" pod "b8b856a0-3adb-4403-a497-91b12ad5d85d" (UID: "b8b856a0-3adb-4403-a497-91b12ad5d85d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:26.129902 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.129866 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.129902 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.129897 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.129902 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.129906 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665kw\" (UniqueName: \"kubernetes.io/projected/b8b856a0-3adb-4403-a497-91b12ad5d85d-kube-api-access-665kw\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.130138 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.129917 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b856a0-3adb-4403-a497-91b12ad5d85d-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:26.353216 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.353179 5693 generic.go:356] "Generic (PLEG): container finished" podID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerID="9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca" exitCode=0 Oct 30 14:00:26.353681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.353251 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" Oct 30 14:00:26.353681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.353259 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" event={"ID":"b8b856a0-3adb-4403-a497-91b12ad5d85d","Type":"ContainerDied","Data":"9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca"} Oct 30 14:00:26.353681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.353330 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4" event={"ID":"b8b856a0-3adb-4403-a497-91b12ad5d85d","Type":"ContainerDied","Data":"6470f7483344d0e0a1b81fd7a26f38c8929e852e8e57ff5b8e60ad31964ee056"} Oct 30 14:00:26.353681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.353362 5693 scope.go:122] "RemoveContainer" containerID="9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca" Oct 30 14:00:26.364841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.364821 5693 scope.go:122] "RemoveContainer" containerID="2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92" Oct 30 14:00:26.375297 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.375280 5693 scope.go:122] "RemoveContainer" containerID="9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca" Oct 30 14:00:26.375567 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:26.375545 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca\": container with ID starting with 9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca not found: ID does not exist" containerID="9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca" Oct 30 14:00:26.375647 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.375575 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca"} err="failed to get container status \"9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca\": rpc error: code = NotFound desc = could not find container \"9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca\": container with ID starting with 9941d544be5a56662d2b3ef1a18bcab37b2dadfba9bfdc30e5dcdb2de1f8dcca not found: ID does not exist" Oct 30 14:00:26.375647 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.375623 5693 scope.go:122] "RemoveContainer" containerID="2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92" Oct 30 14:00:26.375846 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:26.375829 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92\": container with ID starting with 2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92 not found: ID does not exist" containerID="2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92" Oct 30 14:00:26.375891 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.375851 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92"} err="failed to get container status \"2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92\": rpc error: code = NotFound desc = could not find container \"2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92\": container with ID starting with 2e1f1b5174cb077e95dda0b7282c89445f7a3a18d35d62a790272463010afd92 not found: ID does not exist" Oct 30 14:00:26.406737 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.406713 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4"] Oct 30 14:00:26.411106 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:26.411085 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-pdsc4"] Oct 30 14:00:27.489344 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:27.489309 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:28.256879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:28.256845 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" path="/var/lib/kubelet/pods/b8b856a0-3adb-4403-a497-91b12ad5d85d/volumes" Oct 30 14:00:29.126061 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:29.126013 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:30.758617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:30.755504 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:32.506624 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:32.500174 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:33.192109 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:33.192076 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:33.838274 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:33.838219 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:34.152091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:34.151996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:35.799859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:35.799820 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:37.445423 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:37.445262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:39.023661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:39.023537 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:40.640314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:40.640272 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:42.333991 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:42.329488 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:44.015802 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:44.015643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:45.931872 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:45.931813 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:47.849854 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:47.849813 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:49.530404 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:49.530365 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:51.210872 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:51.210837 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:52.602901 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:52.602851 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx"] Oct 30 14:00:52.603435 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:52.603097 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" podUID="8eaa7daf-31f1-45a7-96cd-3064299f6f6a" containerName="maas-api" containerID="cri-o://db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc" gracePeriod=30 Oct 30 14:00:52.707640 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:52.707582 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:52.951981 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:52.951954 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 14:00:53.082476 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.081227 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr\" (UniqueName: \"kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr\") pod \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " Oct 30 14:00:53.082476 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.081282 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca\") pod \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " Oct 30 14:00:53.082476 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.081417 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls\") pod \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " Oct 30 14:00:53.082476 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.081453 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs\") pod \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\" (UID: \"8eaa7daf-31f1-45a7-96cd-3064299f6f6a\") " Oct 30 14:00:53.086358 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.083945 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca" pod "8eaa7daf-31f1-45a7-96cd-3064299f6f6a" (UID: "8eaa7daf-31f1-45a7-96cd-3064299f6f6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:53.096226 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.095828 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs" pod "8eaa7daf-31f1-45a7-96cd-3064299f6f6a" (UID: "8eaa7daf-31f1-45a7-96cd-3064299f6f6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:00:53.128694 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.121253 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr" pod "8eaa7daf-31f1-45a7-96cd-3064299f6f6a" (UID: "8eaa7daf-31f1-45a7-96cd-3064299f6f6a"). InnerVolumeSpecName "kube-api-access-czspr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:53.128694 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:53.126081 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:00:53.129803 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.129766 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls" pod "8eaa7daf-31f1-45a7-96cd-3064299f6f6a" (UID: "8eaa7daf-31f1-45a7-96cd-3064299f6f6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:00:53.187813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.186626 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czspr\" (UniqueName: \"kubernetes.io/projected/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-kube-api-access-czspr\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:53.187813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.186667 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:53.187813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.186682 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:53.187813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.186695 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eaa7daf-31f1-45a7-96cd-3064299f6f6a-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:00:53.270175 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.270132 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:53.431622 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.417705 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4"] Oct 30 14:00:53.451570 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.451468 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c365-v7xr4"] Oct 30 14:00:53.653932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.653899 5693 generic.go:356] "Generic (PLEG): container finished" podID="8eaa7daf-31f1-45a7-96cd-3064299f6f6a" containerID="db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc" exitCode=0 Oct 30 14:00:53.654323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.653946 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" Oct 30 14:00:53.654323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.653986 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" event={"ID":"8eaa7daf-31f1-45a7-96cd-3064299f6f6a","Type":"ContainerDied","Data":"db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc"} Oct 30 14:00:53.654323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.654043 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx" event={"ID":"8eaa7daf-31f1-45a7-96cd-3064299f6f6a","Type":"ContainerDied","Data":"5f7aee99855ec5c3f59385925da5ffb711af8f1d5a599bcb430f6eb2c397df31"} Oct 30 14:00:53.654323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.654073 5693 scope.go:122] "RemoveContainer" containerID="db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc" Oct 30 14:00:53.665468 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.665450 5693 scope.go:122] "RemoveContainer" containerID="db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc" Oct 30 14:00:53.665769 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:00:53.665747 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc\": container with ID starting with db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc not found: ID does not exist" containerID="db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc" Oct 30 14:00:53.665847 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.665776 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc"} err="failed to get container status \"db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc\": rpc error: code = NotFound desc = could not find container \"db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc\": container with ID starting with db7bebff214055e86938977c30c422dd11d9477d83b3df417ccdc903ae0407dc not found: ID does not exist" Oct 30 14:00:53.679270 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.679249 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:53.709040 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.708980 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx"] Oct 30 14:00:53.712455 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.712427 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c365-7bdb8dd789-qkbcx"] Oct 30 14:00:53.740480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.740445 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd"] Oct 30 14:00:53.826016 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.825984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:53.968193 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:53.968110 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:54.260472 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:54.260427 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eaa7daf-31f1-45a7-96cd-3064299f6f6a" path="/var/lib/kubelet/pods/8eaa7daf-31f1-45a7-96cd-3064299f6f6a/volumes" Oct 30 14:00:54.261123 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:54.261096 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91243fe1-ca46-4a14-97f2-821242a8edf9" path="/var/lib/kubelet/pods/91243fe1-ca46-4a14-97f2-821242a8edf9/volumes" Oct 30 14:00:55.250486 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:55.250397 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:56.911859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:56.911816 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:58.168541 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:58.168511 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:58.554582 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:00:58.554524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:00.230818 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:00.230787 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:00.765093 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:00.764987 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:00.920921 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:00.920885 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:01.246745 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:01.246710 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:02.055162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:02.055116 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:03.936153 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:03.936120 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:04.831057 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:04.831024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:04.970381 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:04.970343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:05.416166 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:05.416132 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:07.074421 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:07.074386 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:08.728907 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:08.728872 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:08.768347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:08.768301 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" podUID="431841f6-1652-487c-8170-458c16d4251f" containerName="main" containerID="cri-o://d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491" gracePeriod=45 Oct 30 14:01:09.054390 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.054364 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 14:01:09.103433 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103400 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir\") pod \"431841f6-1652-487c-8170-458c16d4251f\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " Oct 30 14:01:09.103617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103446 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home\") pod \"431841f6-1652-487c-8170-458c16d4251f\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " Oct 30 14:01:09.103617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103489 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache\") pod \"431841f6-1652-487c-8170-458c16d4251f\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " Oct 30 14:01:09.103617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103610 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q\" (UniqueName: \"kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q\") pod \"431841f6-1652-487c-8170-458c16d4251f\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " Oct 30 14:01:09.103823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103656 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm\") pod \"431841f6-1652-487c-8170-458c16d4251f\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " Oct 30 14:01:09.103823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103671 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir" pod "431841f6-1652-487c-8170-458c16d4251f" (UID: "431841f6-1652-487c-8170-458c16d4251f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:09.103823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103770 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home" pod "431841f6-1652-487c-8170-458c16d4251f" (UID: "431841f6-1652-487c-8170-458c16d4251f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:09.103823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103804 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache" pod "431841f6-1652-487c-8170-458c16d4251f" (UID: "431841f6-1652-487c-8170-458c16d4251f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:09.103823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.103809 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs\" (UniqueName: \"kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs\") pod \"431841f6-1652-487c-8170-458c16d4251f\" (UID: \"431841f6-1652-487c-8170-458c16d4251f\") " Oct 30 14:01:09.104744 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.104715 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:09.104744 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.104746 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:09.104958 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.104759 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:09.106069 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.106036 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm" pod "431841f6-1652-487c-8170-458c16d4251f" (UID: "431841f6-1652-487c-8170-458c16d4251f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:09.106188 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.106139 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs" pod "431841f6-1652-487c-8170-458c16d4251f" (UID: "431841f6-1652-487c-8170-458c16d4251f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:01:09.106700 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.106678 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q" pod "431841f6-1652-487c-8170-458c16d4251f" (UID: "431841f6-1652-487c-8170-458c16d4251f"). InnerVolumeSpecName "kube-api-access-nnb8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:01:09.206362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.206331 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnb8q\" (UniqueName: \"kubernetes.io/projected/431841f6-1652-487c-8170-458c16d4251f-kube-api-access-nnb8q\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:09.206362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.206360 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431841f6-1652-487c-8170-458c16d4251f-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:09.206362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.206369 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431841f6-1652-487c-8170-458c16d4251f-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:09.810973 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.810931 5693 generic.go:356] "Generic (PLEG): container finished" podID="431841f6-1652-487c-8170-458c16d4251f" containerID="d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491" exitCode=0 Oct 30 14:01:09.810973 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.810971 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" Oct 30 14:01:09.811511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.811011 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" event={"ID":"431841f6-1652-487c-8170-458c16d4251f","Type":"ContainerDied","Data":"d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491"} Oct 30 14:01:09.811511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.811055 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd" event={"ID":"431841f6-1652-487c-8170-458c16d4251f","Type":"ContainerDied","Data":"01f017b6916b9fec6dd660994bb00c711d75a14527d51ee7c1f65527aff29ee3"} Oct 30 14:01:09.811511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.811076 5693 scope.go:122] "RemoveContainer" containerID="d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491" Oct 30 14:01:09.823096 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.823072 5693 scope.go:122] "RemoveContainer" containerID="d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491" Oct 30 14:01:09.823360 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:01:09.823342 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491\": container with ID starting with d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491 not found: ID does not exist" containerID="d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491" Oct 30 14:01:09.823427 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.823368 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491"} err="failed to get container status \"d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491\": rpc error: code = NotFound desc = could not find container \"d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491\": container with ID starting with d0e8468673d98653efc5aa8ef20e652c1c2d4101d9f50427c3cdfa6e181c0491 not found: ID does not exist" Oct 30 14:01:09.863340 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.863306 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd"] Oct 30 14:01:09.874937 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:09.874903 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-319179/test-model-319179-kserve-65ff788cf5-vwfxd"] Oct 30 14:01:10.258750 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:10.258707 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431841f6-1652-487c-8170-458c16d4251f" path="/var/lib/kubelet/pods/431841f6-1652-487c-8170-458c16d4251f/volumes" Oct 30 14:01:10.616677 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:10.601798 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:12.242752 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:12.242719 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:12.542564 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:12.542465 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:13.918637 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:13.918589 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:15.585619 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:15.579121 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:17.325637 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:17.324762 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:19.022397 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:19.021977 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:20.739641 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:20.739549 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:22.334333 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:22.334295 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:23.058172 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:01:23.058131 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:01:24.025564 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:24.025532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:25.632510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:25.632475 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:27.221691 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:27.221655 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:28.943619 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:28.943143 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:30.708972 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:30.708934 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:32.324400 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:32.324342 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:34.038097 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:34.038057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:35.704011 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:35.703953 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:37.398412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:37.398376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:39.100591 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:39.100550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:40.714000 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:40.713969 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:42.271782 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:42.271723 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:43.928978 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:43.928938 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:45.259648 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:45.259587 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:45.789706 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:45.789672 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:45.934019 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:45.933988 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:46.070386 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:46.070294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:46.461084 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:46.460998 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:47.848882 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:47.848841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:49.349735 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:49.349698 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:49.841241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:49.841200 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:51.145635 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:51.145462 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:52.557570 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:52.557538 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:53.059054 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:01:53.059017 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:01:53.063487 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.063450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:53.226222 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.226193 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:53.379091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.378995 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:53.488002 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.487966 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl"] Oct 30 14:01:53.488538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.488503 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" containerID="cri-o://4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc" gracePeriod=30 Oct 30 14:01:53.563416 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.563375 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:53.712108 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:53.712025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:55.165646 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:55.158845 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:55.858780 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:55.858628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:56.045347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:56.045311 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:56.487728 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:56.487692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:56.935700 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:56.917039 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:58.282982 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.282923 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:58.508202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.508173 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:58.779951 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.779919 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 14:01:58.897398 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897317 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897398 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897365 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897398 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897391 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897738 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897425 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897738 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897544 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897738 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897633 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897708 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:58.897919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897751 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:58.897919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897829 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897901 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket\") pod \"dcbd7a3b-60f4-4623-8b54-345bff163209\" (UID: \"dcbd7a3b-60f4-4623-8b54-345bff163209\") " Oct 30 14:01:58.897919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.897827 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:01:58.898205 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898016 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:58.898279 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898232 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:58.898668 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898643 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.898788 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898671 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.898788 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898685 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.898788 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898700 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbd7a3b-60f4-4623-8b54-345bff163209-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.898788 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.898714 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.899976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.899946 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:01:58.900151 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.899985 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:01:58.900151 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.900061 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:01:58.900151 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.900095 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg" pod "dcbd7a3b-60f4-4623-8b54-345bff163209" (UID: "dcbd7a3b-60f4-4623-8b54-345bff163209"). InnerVolumeSpecName "kube-api-access-7g2rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:01:58.999980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.999943 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.999980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.999972 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.999980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.999981 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2rg\" (UniqueName: \"kubernetes.io/projected/dcbd7a3b-60f4-4623-8b54-345bff163209-kube-api-access-7g2rg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:58.999980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:58.999990 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbd7a3b-60f4-4623-8b54-345bff163209-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:01:59.239414 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.239376 5693 generic.go:356] "Generic (PLEG): container finished" podID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerID="4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc" exitCode=0 Oct 30 14:01:59.241457 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.239477 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" event={"ID":"dcbd7a3b-60f4-4623-8b54-345bff163209","Type":"ContainerDied","Data":"4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc"} Oct 30 14:01:59.241457 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.239581 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" event={"ID":"dcbd7a3b-60f4-4623-8b54-345bff163209","Type":"ContainerDied","Data":"3617cfe623fcfdb28b148190cfd5b38e069d71c0c44b480d57ac9de729d8d1cc"} Oct 30 14:01:59.241457 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.239503 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl" Oct 30 14:01:59.241457 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.239643 5693 scope.go:122] "RemoveContainer" containerID="4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc" Oct 30 14:01:59.252010 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.251986 5693 scope.go:122] "RemoveContainer" containerID="32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f" Oct 30 14:01:59.262102 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.262079 5693 scope.go:122] "RemoveContainer" containerID="4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc" Oct 30 14:01:59.262376 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:01:59.262356 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc\": container with ID starting with 4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc not found: ID does not exist" containerID="4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc" Oct 30 14:01:59.262442 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.262385 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc"} err="failed to get container status \"4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc\": rpc error: code = NotFound desc = could not find container \"4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc\": container with ID starting with 4fc04b95e981faa428a295ed24e8fce0bb0db11573b74d89774d6e878a70dffc not found: ID does not exist" Oct 30 14:01:59.262442 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.262402 5693 scope.go:122] "RemoveContainer" containerID="32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f" Oct 30 14:01:59.262625 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:01:59.262609 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f\": container with ID starting with 32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f not found: ID does not exist" containerID="32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f" Oct 30 14:01:59.262678 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.262628 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f"} err="failed to get container status \"32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f\": rpc error: code = NotFound desc = could not find container \"32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f\": container with ID starting with 32d197819f262a50bf914be5f51e28adcd27df6f1dcec040ac7edc8194b25d0f not found: ID does not exist" Oct 30 14:01:59.291651 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.291614 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl"] Oct 30 14:01:59.303704 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:01:59.303670 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-926ff0-openshift-default-8d578b7c-wxnhl"] Oct 30 14:02:00.118716 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:00.118675 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:00.255027 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:00.254990 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" path="/var/lib/kubelet/pods/dcbd7a3b-60f4-4623-8b54-345bff163209/volumes" Oct 30 14:02:00.391553 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:00.391454 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:00.541153 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:00.541115 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:00.720855 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:00.720771 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:01.668879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:01.668850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:01.817398 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:01.817359 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:02.100012 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:02.099978 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:03.639343 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:03.635495 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:04.912562 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:04.912518 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:05.050318 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:05.050285 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:05.232799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:05.232765 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:06.663944 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:06.663907 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:06.816908 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:06.811736 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:07.423471 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:07.423437 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:07.641590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:07.641526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:08.412537 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:08.412496 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:10.144790 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:10.144749 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:11.772620 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:11.772550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:13.378334 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:13.378302 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:14.313446 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:14.313407 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:15.120200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:15.120150 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:16.773262 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:16.773225 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:18.277375 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.277340 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:18.450962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.450924 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj"] Oct 30 14:02:18.451862 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451836 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" Oct 30 14:02:18.451862 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451860 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451879 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451887 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451900 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451907 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451916 5693 cpu_manager.go:414] "containerMap: removing container" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451924 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451934 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451941 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451950 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451957 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451966 5693 cpu_manager.go:414] "containerMap: removing container" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451974 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451984 5693 cpu_manager.go:414] "containerMap: removing container" podUID="91243fe1-ca46-4a14-97f2-821242a8edf9" containerName="revoke-keys" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.451991 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91243fe1-ca46-4a14-97f2-821242a8edf9" containerName="revoke-keys" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452012 5693 cpu_manager.go:414] "containerMap: removing container" podUID="26296c50-cd6e-4627-891d-705cf55c1b51" containerName="maas-api" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452019 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26296c50-cd6e-4627-891d-705cf55c1b51" containerName="maas-api" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452036 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452043 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452056 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452063 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.452066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452075 5693 cpu_manager.go:414] "containerMap: removing container" podUID="431841f6-1652-487c-8170-458c16d4251f" containerName="main" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452082 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431841f6-1652-487c-8170-458c16d4251f" containerName="main" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452098 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8eaa7daf-31f1-45a7-96cd-3064299f6f6a" containerName="maas-api" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452105 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eaa7daf-31f1-45a7-96cd-3064299f6f6a" containerName="maas-api" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452116 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452123 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452138 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452145 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452181 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452188 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452202 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452209 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452220 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452226 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452235 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452243 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452256 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452262 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452296 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452302 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452315 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452322 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452484 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452497 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452507 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452516 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452525 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452536 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452546 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452560 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452569 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452579 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452590 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452612 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452620 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbd7a3b-60f4-4623-8b54-345bff163209" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452627 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452635 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452643 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452653 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="91243fe1-ca46-4a14-97f2-821242a8edf9" containerName="revoke-keys" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452666 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="26296c50-cd6e-4627-891d-705cf55c1b51" containerName="maas-api" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452677 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452690 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1836005b-46a2-45c3-b269-75bfbf3cabfb" containerName="istio-proxy" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452698 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fecee25-de7d-456b-9b2e-e4907ea32af5" containerName="cleanup" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452706 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="431841f6-1652-487c-8170-458c16d4251f" containerName="main" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452716 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eaa7daf-31f1-45a7-96cd-3064299f6f6a" containerName="maas-api" Oct 30 14:02:18.453361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.452724 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b856a0-3adb-4403-a497-91b12ad5d85d" containerName="istio-proxy" Oct 30 14:02:18.457460 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.457439 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.492187 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.492152 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj"] Oct 30 14:02:18.500538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.500505 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.500885 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.500857 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.501229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.501208 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstx5\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.501416 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.501398 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.501654 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.501630 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.501748 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.501714 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.501931 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.501876 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.502013 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.501952 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.502078 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.502021 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.540642 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.540537 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:18.603621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.603058 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.603621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.603147 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.603890 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.603802 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.603890 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.603850 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.604016 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.603930 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstx5\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.604016 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.603962 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.604127 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.604035 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.604127 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.604109 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.604242 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.604218 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.604303 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.604240 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.605365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.604553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.605365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.604900 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.605365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.605025 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.605365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.605150 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.607639 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.606905 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.609916 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.609886 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.623927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.623896 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.624052 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.623989 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstx5\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5\") pod \"maas-default-gateway-openshift-default-67bf986f99-5w2dj\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:18.780157 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:18.780107 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:19.123459 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.123405 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj"] Oct 30 14:02:19.125954 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:02:19.125927 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffcbe21_629f_402e_9847_02108c1a0392.slice/crio-f203bfc46a2714815cfc380b4c88a5680bf4ee89ac9ef720c811c978e4e811b6 WatchSource:0}: Error finding container f203bfc46a2714815cfc380b4c88a5680bf4ee89ac9ef720c811c978e4e811b6: Status 404 returned error can't find the container with id f203bfc46a2714815cfc380b4c88a5680bf4ee89ac9ef720c811c978e4e811b6 Oct 30 14:02:19.128583 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.128546 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:02:19.128761 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.128643 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:02:19.128761 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.128692 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:02:19.394109 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.394022 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" event={"ID":"1ffcbe21-629f-402e-9847-02108c1a0392","Type":"ContainerStarted","Data":"a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa"} Oct 30 14:02:19.394109 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.394081 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" event={"ID":"1ffcbe21-629f-402e-9847-02108c1a0392","Type":"ContainerStarted","Data":"f203bfc46a2714815cfc380b4c88a5680bf4ee89ac9ef720c811c978e4e811b6"} Oct 30 14:02:19.438694 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.438635 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podStartSLOduration=1.438589642 podStartE2EDuration="1.438589642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:19.435589584 +0000 UTC m=+3515.302207757" watchObservedRunningTime="2026-10-30 14:02:19.438589642 +0000 UTC m=+3515.305207815" Oct 30 14:02:19.780714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:19.780674 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:20.218484 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:20.212073 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:20.782787 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:20.782746 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5w2dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Oct 30 14:02:20.783194 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:20.782800 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Oct 30 14:02:21.781385 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:21.781343 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5w2dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Oct 30 14:02:21.781578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:21.781402 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Oct 30 14:02:21.867622 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:21.860520 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:22.781166 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:22.781122 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5w2dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Oct 30 14:02:22.781344 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:22.781214 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Oct 30 14:02:23.079588 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:02:23.079509 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:02:23.613577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:23.613543 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:23.780419 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:23.780383 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5w2dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Oct 30 14:02:23.780637 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:23.780436 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Oct 30 14:02:23.960746 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:23.960661 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:24.752559 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:24.752524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:24.781010 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:24.780960 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5w2dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Oct 30 14:02:24.781228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:24.781032 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Oct 30 14:02:25.309902 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:25.309863 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:25.780442 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:25.780407 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5w2dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Oct 30 14:02:25.780963 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:25.780458 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Oct 30 14:02:26.942030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:26.941994 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:27.598193 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:27.598155 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:27.598519 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:27.598490 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:27.599670 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:27.599649 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:02:28.619957 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:28.619916 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:30.208797 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:30.206811 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:31.818538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:31.818502 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:33.494811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:33.494772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:35.077450 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:35.077392 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:36.712172 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:36.712111 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:38.326763 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:38.326714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:39.546318 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:39.546280 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt"] Oct 30 14:02:39.547680 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:39.547647 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" podUID="b637c852-682d-40f7-8071-987466336178" containerName="maas-api" containerID="cri-o://13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9" gracePeriod=30 Oct 30 14:02:39.627377 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:39.627352 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:40.005385 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.005345 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 14:02:40.088059 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.088024 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca\") pod \"b637c852-682d-40f7-8071-987466336178\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " Oct 30 14:02:40.088289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.088210 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs\") pod \"b637c852-682d-40f7-8071-987466336178\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " Oct 30 14:02:40.088395 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.088380 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt\" (UniqueName: \"kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt\") pod \"b637c852-682d-40f7-8071-987466336178\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " Oct 30 14:02:40.088467 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.088435 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls\") pod \"b637c852-682d-40f7-8071-987466336178\" (UID: \"b637c852-682d-40f7-8071-987466336178\") " Oct 30 14:02:40.088467 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.088443 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca" pod "b637c852-682d-40f7-8071-987466336178" (UID: "b637c852-682d-40f7-8071-987466336178"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:02:40.089293 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.089261 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b637c852-682d-40f7-8071-987466336178-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:02:40.091560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.091420 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs" pod "b637c852-682d-40f7-8071-987466336178" (UID: "b637c852-682d-40f7-8071-987466336178"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:02:40.091560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.091440 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls" pod "b637c852-682d-40f7-8071-987466336178" (UID: "b637c852-682d-40f7-8071-987466336178"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:02:40.091764 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.091680 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt" pod "b637c852-682d-40f7-8071-987466336178" (UID: "b637c852-682d-40f7-8071-987466336178"). InnerVolumeSpecName "kube-api-access-p2gpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:02:40.131732 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.131665 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:40.190316 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.190286 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:02:40.190316 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.190318 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gpt\" (UniqueName: \"kubernetes.io/projected/b637c852-682d-40f7-8071-987466336178-kube-api-access-p2gpt\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:02:40.190529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.190328 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b637c852-682d-40f7-8071-987466336178-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:02:40.276174 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.276137 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:40.480811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.480774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:40.599174 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.599138 5693 generic.go:356] "Generic (PLEG): container finished" podID="b637c852-682d-40f7-8071-987466336178" containerID="13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9" exitCode=0 Oct 30 14:02:40.599590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.599194 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" Oct 30 14:02:40.599590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.599224 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" event={"ID":"b637c852-682d-40f7-8071-987466336178","Type":"ContainerDied","Data":"13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9"} Oct 30 14:02:40.599590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.599278 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt" event={"ID":"b637c852-682d-40f7-8071-987466336178","Type":"ContainerDied","Data":"1b787a799f98a06014a476ad1f1cc2409128fd0b873c590772230b9d5ddddf90"} Oct 30 14:02:40.599590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.599303 5693 scope.go:122] "RemoveContainer" containerID="13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9" Oct 30 14:02:40.609949 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.609932 5693 scope.go:122] "RemoveContainer" containerID="13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9" Oct 30 14:02:40.610254 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:02:40.610232 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9\": container with ID starting with 13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9 not found: ID does not exist" containerID="13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9" Oct 30 14:02:40.610341 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.610260 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9"} err="failed to get container status \"13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9\": rpc error: code = NotFound desc = could not find container \"13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9\": container with ID starting with 13ce809c8c37a020b3ff9a2d0b7a594bac83479b683e0b3f77fb172a694b26b9 not found: ID does not exist" Oct 30 14:02:40.651902 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.651868 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt"] Oct 30 14:02:40.657128 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.657104 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-319179-c6bfb87f-nklqt"] Oct 30 14:02:40.777208 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:40.777123 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:42.191191 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:42.191156 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:42.256632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:42.256580 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b637c852-682d-40f7-8071-987466336178" path="/var/lib/kubelet/pods/b637c852-682d-40f7-8071-987466336178/volumes" Oct 30 14:02:43.780099 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:43.780024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:45.811506 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:45.811453 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:48.389351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.389315 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:48.546006 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.545942 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:48.607683 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.607645 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9"] Oct 30 14:02:48.608506 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608488 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b637c852-682d-40f7-8071-987466336178" containerName="maas-api" Oct 30 14:02:48.608558 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608510 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b637c852-682d-40f7-8071-987466336178" containerName="maas-api" Oct 30 14:02:48.608558 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608553 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608558 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608584 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608589 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608617 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608624 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9408ace-b372-4dde-8af1-f6f599bc9feb" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608634 5693 cpu_manager.go:414] "containerMap: removing container" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:48.608662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608640 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814dfafd-ce29-4857-829c-56024ac85a80" containerName="cleanup" Oct 30 14:02:48.608888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608747 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80827f5-4b51-42c8-bf30-75ff4c262fd7" containerName="cleanup" Oct 30 14:02:48.608888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608757 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9619f1ee-11f4-4005-96bb-8644b95977e0" containerName="cleanup" Oct 30 14:02:48.608888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608769 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35f61f6-7860-4247-9e3d-b21ad5f66b79" containerName="cleanup" Oct 30 14:02:48.608888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.608776 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b637c852-682d-40f7-8071-987466336178" containerName="maas-api" Oct 30 14:02:48.612711 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.612687 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.639724 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.639647 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9"] Oct 30 14:02:48.718580 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.718538 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.718937 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.718906 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcxl\" (UniqueName: \"kubernetes.io/projected/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-kube-api-access-6qcxl\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719077 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.718951 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719156 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.719073 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719213 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.719167 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719213 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.719201 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719298 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.719276 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719359 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.719341 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.719425 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.719407 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821377 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821341 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821583 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821510 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821583 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821571 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcxl\" (UniqueName: \"kubernetes.io/projected/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-kube-api-access-6qcxl\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821731 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821631 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821731 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821665 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821731 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821709 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821741 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821769 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821814 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.821943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.821844 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.822215 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.822191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.822215 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.822206 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.822336 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.822269 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.822336 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.822310 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.824379 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.824352 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.824772 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.824749 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.840831 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.840805 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcxl\" (UniqueName: \"kubernetes.io/projected/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-kube-api-access-6qcxl\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.841017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.840997 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed2fc6d-db5c-4423-9ae5-0334677b4e1f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rjvw9\" (UID: \"eed2fc6d-db5c-4423-9ae5-0334677b4e1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:48.932537 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:48.932437 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:49.254523 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.253217 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:49.278679 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.278650 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9"] Oct 30 14:02:49.281055 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:02:49.281027 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed2fc6d_db5c_4423_9ae5_0334677b4e1f.slice/crio-4da2817d86b0ff76ef58c2315c73eb710115e42192c780ff4356b1e8958ad7d9 WatchSource:0}: Error finding container 4da2817d86b0ff76ef58c2315c73eb710115e42192c780ff4356b1e8958ad7d9: Status 404 returned error can't find the container with id 4da2817d86b0ff76ef58c2315c73eb710115e42192c780ff4356b1e8958ad7d9 Oct 30 14:02:49.284502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.284452 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:02:49.284655 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.284546 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:02:49.284655 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.284632 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:02:49.519314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.519221 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:49.692614 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.692574 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" event={"ID":"eed2fc6d-db5c-4423-9ae5-0334677b4e1f","Type":"ContainerStarted","Data":"6a55285fc1c7c5b77b10a4aa1f032b664dec50047c988b630bbb1f9046088f3d"} Oct 30 14:02:49.692807 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.692650 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" event={"ID":"eed2fc6d-db5c-4423-9ae5-0334677b4e1f","Type":"ContainerStarted","Data":"4da2817d86b0ff76ef58c2315c73eb710115e42192c780ff4356b1e8958ad7d9"} Oct 30 14:02:49.722293 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.722262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:49.744030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.743976 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podStartSLOduration=1.743962359 podStartE2EDuration="1.743962359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:49.741286167 +0000 UTC m=+3545.607904341" watchObservedRunningTime="2026-10-30 14:02:49.743962359 +0000 UTC m=+3545.610580529" Oct 30 14:02:49.941678 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:49.932683 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:50.664582 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:50.664551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:50.953810 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:50.953715 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:50.954086 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:50.954045 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:51.942624 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:51.937938 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:51.942624 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:51.938011 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:52.358685 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:52.358637 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:52.952037 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:52.951999 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:52.952901 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:52.952651 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:53.081213 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:02:53.081170 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:02:53.369060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:53.369025 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq"] Oct 30 14:02:53.485949 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:53.485909 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:53.666983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:53.666870 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:53.941682 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:53.939951 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:53.941682 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:53.940027 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:54.951782 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:54.938963 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:54.951782 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:54.939045 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:55.448343 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:55.447017 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:55.933472 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:55.933434 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:55.933662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:55.933495 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:56.933735 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:56.933699 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rjvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Oct 30 14:02:56.934133 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:56.933752 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" podUID="eed2fc6d-db5c-4423-9ae5-0334677b4e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Oct 30 14:02:57.196399 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:57.196291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:57.938295 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:57.938258 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:57.938771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:57.938678 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:57.939496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:57.939464 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rjvw9" Oct 30 14:02:58.455242 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:58.455209 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:59.053798 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:02:59.052868 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:00.887473 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:00.887439 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:02.512384 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:02.512203 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:03.311048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:03.310990 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:04.266620 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:04.262962 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:05.937138 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:05.937095 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:07.526268 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:07.526114 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:08.392034 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.391999 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" podUID="6a546dd6-cef6-4d84-835c-89464b7a63b6" containerName="main" containerID="cri-o://2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051" gracePeriod=45 Oct 30 14:03:08.673804 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.673773 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 14:03:08.777030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.776998 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29\" (UniqueName: \"kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29\") pod \"6a546dd6-cef6-4d84-835c-89464b7a63b6\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " Oct 30 14:03:08.777228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777050 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm\") pod \"6a546dd6-cef6-4d84-835c-89464b7a63b6\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " Oct 30 14:03:08.777228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777187 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir\") pod \"6a546dd6-cef6-4d84-835c-89464b7a63b6\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " Oct 30 14:03:08.777383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777311 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs\") pod \"6a546dd6-cef6-4d84-835c-89464b7a63b6\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " Oct 30 14:03:08.777383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777365 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache\") pod \"6a546dd6-cef6-4d84-835c-89464b7a63b6\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " Oct 30 14:03:08.777588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777567 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home\") pod \"6a546dd6-cef6-4d84-835c-89464b7a63b6\" (UID: \"6a546dd6-cef6-4d84-835c-89464b7a63b6\") " Oct 30 14:03:08.777817 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777783 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache" pod "6a546dd6-cef6-4d84-835c-89464b7a63b6" (UID: "6a546dd6-cef6-4d84-835c-89464b7a63b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:03:08.777899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.777803 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir" pod "6a546dd6-cef6-4d84-835c-89464b7a63b6" (UID: "6a546dd6-cef6-4d84-835c-89464b7a63b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:03:08.778086 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.778060 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home" pod "6a546dd6-cef6-4d84-835c-89464b7a63b6" (UID: "6a546dd6-cef6-4d84-835c-89464b7a63b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:03:08.778657 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.778629 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:08.778757 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.778660 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:08.778757 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.778676 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:08.779593 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.779568 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm" pod "6a546dd6-cef6-4d84-835c-89464b7a63b6" (UID: "6a546dd6-cef6-4d84-835c-89464b7a63b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:03:08.779827 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.779808 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29" pod "6a546dd6-cef6-4d84-835c-89464b7a63b6" (UID: "6a546dd6-cef6-4d84-835c-89464b7a63b6"). InnerVolumeSpecName "kube-api-access-qxl29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:03:08.780217 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.780200 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs" pod "6a546dd6-cef6-4d84-835c-89464b7a63b6" (UID: "6a546dd6-cef6-4d84-835c-89464b7a63b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:03:08.879950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.879915 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a546dd6-cef6-4d84-835c-89464b7a63b6-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:08.879950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.879946 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxl29\" (UniqueName: \"kubernetes.io/projected/6a546dd6-cef6-4d84-835c-89464b7a63b6-kube-api-access-qxl29\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:08.879950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.879956 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a546dd6-cef6-4d84-835c-89464b7a63b6-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:08.893050 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.893014 5693 generic.go:356] "Generic (PLEG): container finished" podID="6a546dd6-cef6-4d84-835c-89464b7a63b6" containerID="2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051" exitCode=0 Oct 30 14:03:08.893225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.893065 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" Oct 30 14:03:08.893225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.893097 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" event={"ID":"6a546dd6-cef6-4d84-835c-89464b7a63b6","Type":"ContainerDied","Data":"2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051"} Oct 30 14:03:08.893225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.893142 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq" event={"ID":"6a546dd6-cef6-4d84-835c-89464b7a63b6","Type":"ContainerDied","Data":"48d78f1a5e42304181672bae350241b4cdc1577914cbcf5601c5aacf61a26800"} Oct 30 14:03:08.893225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.893162 5693 scope.go:122] "RemoveContainer" containerID="2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051" Oct 30 14:03:08.905015 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.904993 5693 scope.go:122] "RemoveContainer" containerID="2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051" Oct 30 14:03:08.905278 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:03:08.905258 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051\": container with ID starting with 2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051 not found: ID does not exist" containerID="2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051" Oct 30 14:03:08.905341 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.905286 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051"} err="failed to get container status \"2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051\": rpc error: code = NotFound desc = could not find container \"2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051\": container with ID starting with 2660ee3acd697de2c0cabc4d2810187b653d7fd78701067b52c15780f7fbd051 not found: ID does not exist" Oct 30 14:03:08.952570 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.952530 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq"] Oct 30 14:03:08.957894 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:08.957863 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d724c8/test-model-d724c8-kserve-68cb5d6999-2hvcq"] Oct 30 14:03:09.349234 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:09.349195 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:10.255769 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:10.255733 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a546dd6-cef6-4d84-835c-89464b7a63b6" path="/var/lib/kubelet/pods/6a546dd6-cef6-4d84-835c-89464b7a63b6/volumes" Oct 30 14:03:10.716288 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:10.716213 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:10.937105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:10.937050 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:12.428564 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:12.428509 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:12.593619 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:12.584774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:15.059506 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:15.059458 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:17.587841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:17.587798 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:19.298845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:19.298809 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:20.172781 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.172743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:20.566264 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.566225 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs"] Oct 30 14:03:20.567525 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.567497 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6a546dd6-cef6-4d84-835c-89464b7a63b6" containerName="main" Oct 30 14:03:20.567525 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.567521 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a546dd6-cef6-4d84-835c-89464b7a63b6" containerName="main" Oct 30 14:03:20.567746 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.567700 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a546dd6-cef6-4d84-835c-89464b7a63b6" containerName="main" Oct 30 14:03:20.572146 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.572120 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.589747 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.589714 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9b32c32a-openshift-default-dockercfg-f77tj\"" Oct 30 14:03:20.607042 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.607009 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs"] Oct 30 14:03:20.654786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.654750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.654786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.654787 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.654999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.654819 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jpq\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.654999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.654966 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.655070 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.655052 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.655164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.655144 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.655236 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.655219 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.655335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.655316 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.655370 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.655348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.752855 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.752818 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.764288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.764460 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.764491 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765058 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765101 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765228 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765378 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jpq\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765413 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765633 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.765705 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.766149 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.766344 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.767633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.767132 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.797621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.776897 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.800335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.799045 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.800729 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.800702 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jpq\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.810643 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.810615 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token\") pod \"e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.889939 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.889833 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:20.963685 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:20.962524 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:21.242123 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:03:21.242092 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b25af17_7179_427d_b5f8_df3fd9f434b7.slice/crio-bc00e3c15a1f83d52baa54f4af5c721124ae04fca131558f656cb493b01b00a3 WatchSource:0}: Error finding container bc00e3c15a1f83d52baa54f4af5c721124ae04fca131558f656cb493b01b00a3: Status 404 returned error can't find the container with id bc00e3c15a1f83d52baa54f4af5c721124ae04fca131558f656cb493b01b00a3 Oct 30 14:03:21.243405 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:21.243371 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs"] Oct 30 14:03:21.245777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:21.245739 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:21.245887 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:21.245825 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:21.245887 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:21.245868 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:21.407134 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:21.407093 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:22.020282 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:22.020239 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" event={"ID":"5b25af17-7179-427d-b5f8-df3fd9f434b7","Type":"ContainerStarted","Data":"ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a"} Oct 30 14:03:22.020703 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:22.020306 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" event={"ID":"5b25af17-7179-427d-b5f8-df3fd9f434b7","Type":"ContainerStarted","Data":"bc00e3c15a1f83d52baa54f4af5c721124ae04fca131558f656cb493b01b00a3"} Oct 30 14:03:22.111804 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:22.111754 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podStartSLOduration=2.1117395 podStartE2EDuration="2.1117395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:22.108458128 +0000 UTC m=+3577.975076301" watchObservedRunningTime="2026-10-30 14:03:22.1117395 +0000 UTC m=+3577.978357757" Oct 30 14:03:22.890088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:22.890057 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:22.896392 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:22.896362 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:23.029027 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:23.029000 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:23.031419 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:23.031397 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:03:23.063226 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:03:23.063190 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:03:23.124298 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:23.124253 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq"] Oct 30 14:03:23.352091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:23.352059 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:23.547898 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:23.547862 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:24.734344 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:24.734297 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:25.038937 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:25.038854 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:25.705789 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:25.705754 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:26.584672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:26.584634 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:26.829585 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:26.829527 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:27.812834 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:27.812799 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:28.124847 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:28.124746 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" containerID="cri-o://52f5eb0d71680ddbde40e0d843163672826c4652e716e9cf2a22f1e38664d6b1" gracePeriod=25 Oct 30 14:03:28.148671 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:28.148634 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:03:29.692866 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:29.692830 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:31.564846 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:31.564800 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:33.810757 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:33.810722 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:35.550302 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:35.550257 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:37.324800 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:37.324765 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:37.865897 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:37.865860 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:38.164210 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:38.159699 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:03:38.198476 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:38.198369 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:38.457151 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:38.457012 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:39.665962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:39.665910 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:41.568938 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:41.568904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:43.330626 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:43.330574 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:45.260383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:45.260334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:45.972378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:45.972343 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:46.147002 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:46.146967 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:46.957091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:46.957048 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:48.147252 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:03:48.147227 5693 logging.go:55] [core] [Channel #17786 SubChannel #17787]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.11:50052", ServerName: "10.134.1.11:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.11:50052: connect: connection refused" Oct 30 14:03:48.383917 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.383669 5693 generic.go:356] "Generic (PLEG): container finished" podID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerID="52f5eb0d71680ddbde40e0d843163672826c4652e716e9cf2a22f1e38664d6b1" exitCode=0 Oct 30 14:03:48.383917 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.383790 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" event={"ID":"dbe33bfc-c4f4-4286-b315-abb0b341bc3e","Type":"ContainerDied","Data":"52f5eb0d71680ddbde40e0d843163672826c4652e716e9cf2a22f1e38664d6b1"} Oct 30 14:03:48.440816 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.440763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:48.448681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.448653 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 14:03:48.529516 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.529474 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume\") pod \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " Oct 30 14:03:48.529750 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.529523 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb\" (UniqueName: \"kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb\") pod \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\" (UID: \"dbe33bfc-c4f4-4286-b315-abb0b341bc3e\") " Oct 30 14:03:48.529976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.529946 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume" pod "dbe33bfc-c4f4-4286-b315-abb0b341bc3e" (UID: "dbe33bfc-c4f4-4286-b315-abb0b341bc3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:03:48.530233 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.530204 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:48.531829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.531797 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb" pod "dbe33bfc-c4f4-4286-b315-abb0b341bc3e" (UID: "dbe33bfc-c4f4-4286-b315-abb0b341bc3e"). InnerVolumeSpecName "kube-api-access-kzjdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:03:48.625122 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.625085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:48.632010 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.631985 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjdb\" (UniqueName: \"kubernetes.io/projected/dbe33bfc-c4f4-4286-b315-abb0b341bc3e-kube-api-access-kzjdb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:03:48.810439 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.810408 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:48.972802 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:48.972764 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:49.147253 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.147153 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.11:50052\" within 1s: context deadline exceeded" Oct 30 14:03:49.147745 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.147261 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 14:03:49.261073 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.261037 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:49.432658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.429763 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" event={"ID":"dbe33bfc-c4f4-4286-b315-abb0b341bc3e","Type":"ContainerDied","Data":"0f1dec7aaea8102803c24469282f3bdcc15590629c0fab73d40ac9849cd32faa"} Oct 30 14:03:49.432658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.429847 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq" Oct 30 14:03:49.432658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.429855 5693 scope.go:122] "RemoveContainer" containerID="52f5eb0d71680ddbde40e0d843163672826c4652e716e9cf2a22f1e38664d6b1" Oct 30 14:03:49.496120 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.496086 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq"] Oct 30 14:03:49.508187 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:49.508154 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d724c8-5b9fd7c444-kf6pq"] Oct 30 14:03:50.256067 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:50.256028 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" path="/var/lib/kubelet/pods/dbe33bfc-c4f4-4286-b315-abb0b341bc3e/volumes" Oct 30 14:03:51.127464 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:51.127429 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:52.817779 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:52.817741 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:53.056413 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:03:53.056377 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:03:54.455467 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:54.455416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:56.219968 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:56.219890 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:56.658896 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:56.658868 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:57.006217 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:57.006183 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:58.358091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:58.358040 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:58.575425 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:03:58.575391 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:00.266570 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:00.266530 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:02.138285 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:02.138247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:03.933399 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:03.933360 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:05.607804 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:05.607611 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:07.260626 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:07.260064 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:07.449831 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:07.449798 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:07.605661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:07.605560 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:08.756114 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:08.756076 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:09.368708 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:09.368674 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:09.524045 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:09.524012 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:09.693055 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:09.692962 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:09.992031 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:09.992001 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:11.666090 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:11.666056 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:13.458208 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:13.458173 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:13.859841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:13.859804 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d"] Oct 30 14:04:13.860069 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:13.860019 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" podUID="a64fe897-44aa-40ff-a273-5c40d458c4a2" containerName="discovery" containerID="cri-o://46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26" gracePeriod=30 Oct 30 14:04:14.163807 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.163780 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 14:04:14.216940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.216908 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.217136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.216999 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.217136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.217031 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.217136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.217090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.217136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.217116 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.217414 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.217172 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.217696 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.217649 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:04:14.217983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.217961 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert\") pod \"a64fe897-44aa-40ff-a273-5c40d458c4a2\" (UID: \"a64fe897-44aa-40ff-a273-5c40d458c4a2\") " Oct 30 14:04:14.220829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.220382 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:04:14.220966 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.220663 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "kube-api-access-tmc49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:04:14.220966 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.220708 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:14.220966 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.220769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:04:14.221416 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221385 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:04:14.221770 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221745 5693 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-ca-configmap\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.221886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221785 5693 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-csr-dns-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.221886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221798 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmc49\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-kube-api-access-tmc49\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.221886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221815 5693 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-kubeconfig\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.221886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221827 5693 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64fe897-44aa-40ff-a273-5c40d458c4a2-local-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.221886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.221840 5693 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a64fe897-44aa-40ff-a273-5c40d458c4a2-cacerts\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.222285 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.222259 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token" pod "a64fe897-44aa-40ff-a273-5c40d458c4a2" (UID: "a64fe897-44aa-40ff-a273-5c40d458c4a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:04:14.323888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.323846 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64fe897-44aa-40ff-a273-5c40d458c4a2-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:14.525239 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.525205 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx"] Oct 30 14:04:14.526894 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.526855 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a64fe897-44aa-40ff-a273-5c40d458c4a2" containerName="discovery" Oct 30 14:04:14.527074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.527057 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64fe897-44aa-40ff-a273-5c40d458c4a2" containerName="discovery" Oct 30 14:04:14.527232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.527210 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" Oct 30 14:04:14.527232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.527227 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" Oct 30 14:04:14.527479 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.527427 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe33bfc-c4f4-4286-b315-abb0b341bc3e" containerName="payload-processing" Oct 30 14:04:14.527479 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.527459 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64fe897-44aa-40ff-a273-5c40d458c4a2" containerName="discovery" Oct 30 14:04:14.531843 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.531824 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.548029 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.548006 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"openshift-service-ca.crt\"" Oct 30 14:04:14.548268 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.548245 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"kube-root-ca.crt\"" Oct 30 14:04:14.554140 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.554112 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx"] Oct 30 14:04:14.559533 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.559512 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"default-dockercfg-kcgpd\"" Oct 30 14:04:14.559663 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.559531 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"sub-test-model-22af91-kserve-self-signed-certs\"" Oct 30 14:04:14.629551 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.629515 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.629799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.629777 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.629895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.629826 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.629985 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.629964 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.630049 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630020 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.630129 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccxx\" (UniqueName: \"kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.630488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630465 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:14.630586 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630506 5693 generic.go:356] "Generic (PLEG): container finished" podID="a64fe897-44aa-40ff-a273-5c40d458c4a2" containerID="46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26" exitCode=0 Oct 30 14:04:14.630586 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630558 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" Oct 30 14:04:14.630728 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630588 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" event={"ID":"a64fe897-44aa-40ff-a273-5c40d458c4a2","Type":"ContainerDied","Data":"46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26"} Oct 30 14:04:14.630728 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630657 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d" event={"ID":"a64fe897-44aa-40ff-a273-5c40d458c4a2","Type":"ContainerDied","Data":"55f7b5f245d8b6c29ecd5577c58c74713c1be42991b8a431685577aac8223cce"} Oct 30 14:04:14.630728 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.630681 5693 scope.go:122] "RemoveContainer" containerID="46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26" Oct 30 14:04:14.648347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.648325 5693 scope.go:122] "RemoveContainer" containerID="46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26" Oct 30 14:04:14.648661 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:14.648636 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26\": container with ID starting with 46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26 not found: ID does not exist" containerID="46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26" Oct 30 14:04:14.648775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.648667 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26"} err="failed to get container status \"46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26\": rpc error: code = NotFound desc = could not find container \"46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26\": container with ID starting with 46017b8ab0e433e870073b9e5026adccc8a81e6ff240597ef26bb1f957219d26 not found: ID does not exist" Oct 30 14:04:14.688739 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.688697 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d"] Oct 30 14:04:14.693906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.693873 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tr48d"] Oct 30 14:04:14.732118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732075 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732307 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732284 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732335 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732434 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732419 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732495 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732466 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732549 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732496 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccxx\" (UniqueName: \"kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732549 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732520 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732652 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.732897 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.732860 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.734663 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.734637 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.735332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.735313 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.756824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.756787 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccxx\" (UniqueName: \"kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx\") pod \"sub-test-model-22af91-kserve-68648b8c7f-f2jbx\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:14.847370 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:14.847280 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:15.047726 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.047689 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj"] Oct 30 14:04:15.052770 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.052747 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.070004 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.069969 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9b32c32a-serving-cert\"" Oct 30 14:04:15.074219 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.074194 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj"] Oct 30 14:04:15.140076 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.139978 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.140255 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.140151 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.140332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.140271 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.140443 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.140417 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27sq\" (UniqueName: \"kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.167184 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.167158 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx"] Oct 30 14:04:15.169354 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:04:15.169325 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfedb5b6c_e1f5_4fd6_a8e3_dcdb40aa42bb.slice/crio-0359b6fd10cb68f8d251afa40285429c0b6a1508735e0e32c61f6140b1c70377 WatchSource:0}: Error finding container 0359b6fd10cb68f8d251afa40285429c0b6a1508735e0e32c61f6140b1c70377: Status 404 returned error can't find the container with id 0359b6fd10cb68f8d251afa40285429c0b6a1508735e0e32c61f6140b1c70377 Oct 30 14:04:15.184698 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.184671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:15.241877 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.241845 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.241998 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.241957 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.242151 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.242046 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27sq\" (UniqueName: \"kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.242225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.242192 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.242711 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.242551 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.244658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.244636 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.244658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.244648 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.264678 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.264652 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27sq\" (UniqueName: \"kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq\") pod \"maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.368905 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.368866 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:15.387066 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.387029 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:15.639834 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.639797 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" event={"ID":"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb","Type":"ContainerStarted","Data":"b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248"} Oct 30 14:04:15.640252 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.639847 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" event={"ID":"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb","Type":"ContainerStarted","Data":"0359b6fd10cb68f8d251afa40285429c0b6a1508735e0e32c61f6140b1c70377"} Oct 30 14:04:15.687743 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.687713 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj"] Oct 30 14:04:15.690222 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:04:15.690197 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74069549_7659_4791_8965_836da15e510c.slice/crio-a1e60990c954c378e7d7239f0374615bfef319e6048d5267a9b81345820aa2fa WatchSource:0}: Error finding container a1e60990c954c378e7d7239f0374615bfef319e6048d5267a9b81345820aa2fa: Status 404 returned error can't find the container with id a1e60990c954c378e7d7239f0374615bfef319e6048d5267a9b81345820aa2fa Oct 30 14:04:15.690370 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.690328 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" podStartSLOduration=1.6903166170000001 podStartE2EDuration="1.690316617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:15.685147653 +0000 UTC m=+3631.551765827" watchObservedRunningTime="2026-10-30 14:04:15.690316617 +0000 UTC m=+3631.556934790" Oct 30 14:04:15.691719 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.691701 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:04:15.847710 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:15.847677 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:16.326958 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:16.326903 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64fe897-44aa-40ff-a273-5c40d458c4a2" path="/var/lib/kubelet/pods/a64fe897-44aa-40ff-a273-5c40d458c4a2/volumes" Oct 30 14:04:16.650014 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:16.649978 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" event={"ID":"74069549-7659-4791-8965-836da15e510c","Type":"ContainerStarted","Data":"650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2"} Oct 30 14:04:16.650442 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:16.650040 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" event={"ID":"74069549-7659-4791-8965-836da15e510c","Type":"ContainerStarted","Data":"a1e60990c954c378e7d7239f0374615bfef319e6048d5267a9b81345820aa2fa"} Oct 30 14:04:16.691950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:16.691893 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" podStartSLOduration=1.245070587 podStartE2EDuration="1.691877392s" totalImagesPullingTime="446.806805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:16.690234314 +0000 UTC m=+3632.556852488" watchObservedRunningTime="2026-10-30 14:04:16.691877392 +0000 UTC m=+3632.558495564" Oct 30 14:04:16.997411 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:16.997364 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:17.656742 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:17.656705 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:18.675962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:18.675926 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:18.873073 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:18.873033 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:19.003273 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:19.003232 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb"] Oct 30 14:04:19.003648 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:19.003590 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" containerID="cri-o://3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67" gracePeriod=30 Oct 30 14:04:19.237962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:19.237912 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:20.783305 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:20.783238 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:22.755835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:22.755795 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:23.056550 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:23.056467 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:04:23.670169 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:23.670140 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:04:23.687144 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:23.687111 5693 patch_prober.go:29] interesting pod/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:04:23.687307 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:23.687176 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:04:23.791364 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:23.791324 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:23.933791 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:23.933712 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.296635 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.296591 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 14:04:24.399835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.399792 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400150 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.399935 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400150 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400027 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400150 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400135 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400356 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400165 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400356 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400215 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:04:24.400356 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400273 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400359 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400387 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.400530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400429 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs\") pod \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\" (UID: \"d851c6f3-ffa9-4084-aa00-dff5be92cb1b\") " Oct 30 14:04:24.401012 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400873 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:24.401012 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.400976 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:24.401241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.401119 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:24.401241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.401189 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.401241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.401210 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.401241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.401226 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.401488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.401390 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:24.402960 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.402928 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:04:24.403352 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.403328 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:04:24.404182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.404153 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:24.404500 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.404475 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg" pod "d851c6f3-ffa9-4084-aa00-dff5be92cb1b" (UID: "d851c6f3-ffa9-4084-aa00-dff5be92cb1b"). InnerVolumeSpecName "kube-api-access-kwklg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:04:24.502441 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.502406 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.502441 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.502437 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.502441 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.502447 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.502707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.502455 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.502707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.502462 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.502707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.502470 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwklg\" (UniqueName: \"kubernetes.io/projected/d851c6f3-ffa9-4084-aa00-dff5be92cb1b-kube-api-access-kwklg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:24.512231 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.512162 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.712756 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.712722 5693 generic.go:356] "Generic (PLEG): container finished" podID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerID="3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67" exitCode=0 Oct 30 14:04:24.712977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.712776 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" Oct 30 14:04:24.712977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.712805 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" event={"ID":"d851c6f3-ffa9-4084-aa00-dff5be92cb1b","Type":"ContainerDied","Data":"3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67"} Oct 30 14:04:24.712977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.712863 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb" event={"ID":"d851c6f3-ffa9-4084-aa00-dff5be92cb1b","Type":"ContainerDied","Data":"d97c952d143896b944fc64fc59975776a5adbcb38f3e497f099e2ef7cd4ef984"} Oct 30 14:04:24.712977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.712888 5693 scope.go:122] "RemoveContainer" containerID="3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67" Oct 30 14:04:24.724856 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.724839 5693 scope.go:122] "RemoveContainer" containerID="ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca" Oct 30 14:04:24.735070 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.735051 5693 scope.go:122] "RemoveContainer" containerID="3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67" Oct 30 14:04:24.735357 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:24.735336 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67\": container with ID starting with 3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67 not found: ID does not exist" containerID="3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67" Oct 30 14:04:24.735410 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.735363 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67"} err="failed to get container status \"3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67\": rpc error: code = NotFound desc = could not find container \"3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67\": container with ID starting with 3322bf75ea856d03cec6fca8aff48a333f4454c58612b96c06d5d12a8ac0be67 not found: ID does not exist" Oct 30 14:04:24.735410 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.735379 5693 scope.go:122] "RemoveContainer" containerID="ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca" Oct 30 14:04:24.735695 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:24.735671 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca\": container with ID starting with ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca not found: ID does not exist" containerID="ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca" Oct 30 14:04:24.735802 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.735704 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca"} err="failed to get container status \"ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca\": rpc error: code = NotFound desc = could not find container \"ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca\": container with ID starting with ca1c8af572194d87b06d4e48967e8d048dccff29af789139d93437c309c366ca not found: ID does not exist" Oct 30 14:04:24.765947 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.765852 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb"] Oct 30 14:04:24.776982 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.776947 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d724c8-openshift-default-6cbf896ddc-rhstb"] Oct 30 14:04:24.847876 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.847840 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:24.860643 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:24.860593 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:25.732731 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:25.732702 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:04:26.193407 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:26.193310 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:26.266924 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:26.266888 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" path="/var/lib/kubelet/pods/d851c6f3-ffa9-4084-aa00-dff5be92cb1b/volumes" Oct 30 14:04:28.003842 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:28.003806 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:29.656129 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:29.656086 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:30.189318 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:30.189276 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:30.326706 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:30.326663 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:31.366139 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:31.366102 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:33.296996 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.296961 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:33.955118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.955071 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9"] Oct 30 14:04:33.956309 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.956279 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" Oct 30 14:04:33.956309 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.956308 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" Oct 30 14:04:33.956531 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.956350 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" Oct 30 14:04:33.956531 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.956359 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" Oct 30 14:04:33.956674 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.956575 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" Oct 30 14:04:33.956674 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.956592 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d851c6f3-ffa9-4084-aa00-dff5be92cb1b" containerName="istio-proxy" Oct 30 14:04:33.961761 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.961740 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.972613 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.972565 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.972789 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.972765 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.972858 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.972796 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.972920 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.972881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.972983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.972968 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.973043 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.973015 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.973043 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.973036 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.973167 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.973147 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.973399 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.973371 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2gb\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:33.976525 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.976490 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9"] Oct 30 14:04:33.983921 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:33.983899 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c684d2-openshift-default-dockercfg-f677f\"" Oct 30 14:04:34.074788 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074750 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.074967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074821 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.074967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074843 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.074967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074864 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.074967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074933 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.074967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074953 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2gb\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.074978 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075127 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075154 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075226 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075364 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075544 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075512 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075638 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075524 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.075793 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.075774 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.077302 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.077282 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.077818 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.077796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.097870 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.097838 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2gb\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.098266 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.098248 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.278101 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.278062 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:34.623626 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.623585 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9"] Oct 30 14:04:34.625465 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:04:34.625440 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394b55a_96df_4246_86b1_bd9d3f98eb18.slice/crio-e97bb85b537f7890b92009a7b8e54e0ddf94a83df7bdc393eb15fd834e3481f8 WatchSource:0}: Error finding container e97bb85b537f7890b92009a7b8e54e0ddf94a83df7bdc393eb15fd834e3481f8: Status 404 returned error can't find the container with id e97bb85b537f7890b92009a7b8e54e0ddf94a83df7bdc393eb15fd834e3481f8 Oct 30 14:04:34.653405 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.653363 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:34.653545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.653469 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:34.653545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.653501 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:34.858520 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.857327 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" event={"ID":"a394b55a-96df-4246-86b1-bd9d3f98eb18","Type":"ContainerStarted","Data":"575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949"} Oct 30 14:04:34.858520 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.857395 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" event={"ID":"a394b55a-96df-4246-86b1-bd9d3f98eb18","Type":"ContainerStarted","Data":"e97bb85b537f7890b92009a7b8e54e0ddf94a83df7bdc393eb15fd834e3481f8"} Oct 30 14:04:34.905027 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.904921 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podStartSLOduration=1.904902791 podStartE2EDuration="1.904902791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:34.901858842 +0000 UTC m=+3650.768477009" watchObservedRunningTime="2026-10-30 14:04:34.904902791 +0000 UTC m=+3650.771520961" Oct 30 14:04:34.906365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:34.906329 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:35.283639 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:35.282020 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:35.284793 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:35.284761 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:35.284968 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:35.284818 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:35.490622 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:35.489674 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:35.658682 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:35.658201 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:36.150281 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:36.150204 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:36.279515 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:36.279474 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:36.279786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:36.279539 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:37.280728 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:37.280691 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:37.281541 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:37.281510 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:37.863732 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:37.849548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:38.280260 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:38.280225 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:38.280579 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:38.280550 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:39.279994 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:39.279956 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:39.280387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:39.280024 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:39.864044 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:39.864004 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:40.279279 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:40.279236 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:40.279473 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:40.279324 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:41.279048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:41.279004 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 14:04:41.279660 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:41.279080 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 14:04:41.636040 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:41.635911 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:42.283648 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:42.283613 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:42.284097 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:42.284078 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:42.284904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:42.284875 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:04:43.297623 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.297578 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:43.466341 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.466240 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:43.571672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.571638 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp"] Oct 30 14:04:43.571874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.571838 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" containerID="cri-o://0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07" gracePeriod=30 Oct 30 14:04:43.645238 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.645205 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:43.949933 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.949899 5693 generic.go:356] "Generic (PLEG): container finished" podID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerID="ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a" exitCode=0 Oct 30 14:04:43.950154 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.949978 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" event={"ID":"5b25af17-7179-427d-b5f8-df3fd9f434b7","Type":"ContainerDied","Data":"ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a"} Oct 30 14:04:43.950499 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.950478 5693 scope.go:122] "RemoveContainer" containerID="ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a" Oct 30 14:04:43.952301 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.952271 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:43.952386 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.952340 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:43.952386 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:43.952372 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:44.338315 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:44.338284 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:44.963280 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:44.963241 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" event={"ID":"5b25af17-7179-427d-b5f8-df3fd9f434b7","Type":"ContainerStarted","Data":"5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333"} Oct 30 14:04:45.215641 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:45.215508 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:45.754015 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:45.753978 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:04:45.754527 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:45.754046 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:04:45.890327 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:45.890285 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:04:45.890655 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:45.890630 5693 patch_prober.go:29] interesting pod/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Oct 30 14:04:45.890720 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:45.890689 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Oct 30 14:04:46.891118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:46.891084 5693 patch_prober.go:29] interesting pod/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Oct 30 14:04:46.891577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:46.891137 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Oct 30 14:04:47.353284 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:47.353207 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:47.890943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:47.890909 5693 patch_prober.go:29] interesting pod/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Oct 30 14:04:47.891126 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:47.890966 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Oct 30 14:04:48.890240 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.890208 5693 patch_prober.go:29] interesting pod/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Oct 30 14:04:48.890621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.890264 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Oct 30 14:04:48.893625 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.893574 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 14:04:48.959953 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.959889 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:48.998693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.997684 5693 generic.go:356] "Generic (PLEG): container finished" podID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerID="0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07" exitCode=0 Oct 30 14:04:48.998693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.997909 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" event={"ID":"75f4286a-6f10-46bc-960b-ae6b3178f744","Type":"ContainerDied","Data":"0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07"} Oct 30 14:04:48.998693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.997968 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" event={"ID":"75f4286a-6f10-46bc-960b-ae6b3178f744","Type":"ContainerDied","Data":"3290b435d57a867745021984db806beb20c3a3b10184981ec8b9395de901ccb0"} Oct 30 14:04:48.998693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.998001 5693 scope.go:122] "RemoveContainer" containerID="0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07" Oct 30 14:04:48.998693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:48.998179 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp" Oct 30 14:04:49.005261 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005197 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005395 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005372 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005447 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005414 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005500 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005483 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005576 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005560 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005662 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005645 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005715 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005701 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005787 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005773 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.005890 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.005877 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert\") pod \"75f4286a-6f10-46bc-960b-ae6b3178f744\" (UID: \"75f4286a-6f10-46bc-960b-ae6b3178f744\") " Oct 30 14:04:49.007672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.007127 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:04:49.007672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.007237 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:49.007672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.007368 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:49.008436 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.008081 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:49.008436 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.008190 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:49.011969 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.011923 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:04:49.014817 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.014789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "kube-api-access-hszvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:04:49.014953 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.014871 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:04:49.015002 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.014987 5693 scope.go:122] "RemoveContainer" containerID="13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23" Oct 30 14:04:49.015833 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.015789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy" pod "75f4286a-6f10-46bc-960b-ae6b3178f744" (UID: "75f4286a-6f10-46bc-960b-ae6b3178f744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:04:49.074524 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.074254 5693 scope.go:122] "RemoveContainer" containerID="0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07" Oct 30 14:04:49.074970 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:49.074944 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07\": container with ID starting with 0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07 not found: ID does not exist" containerID="0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07" Oct 30 14:04:49.075103 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.074982 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07"} err="failed to get container status \"0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07\": rpc error: code = NotFound desc = could not find container \"0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07\": container with ID starting with 0c2758fe1358c236f87543a4b91a724406829602f7ca993a44007fd52540ad07 not found: ID does not exist" Oct 30 14:04:49.075103 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.075007 5693 scope.go:122] "RemoveContainer" containerID="13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23" Oct 30 14:04:49.075394 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:49.075364 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23\": container with ID starting with 13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23 not found: ID does not exist" containerID="13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23" Oct 30 14:04:49.075529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.075503 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23"} err="failed to get container status \"13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23\": rpc error: code = NotFound desc = could not find container \"13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23\": container with ID starting with 13ca7153019331506cfc90b6fa01200065500fa0b2754643e8d11a830f338f23 not found: ID does not exist" Oct 30 14:04:49.108188 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108153 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hszvz\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-kube-api-access-hszvz\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108188 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108186 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108197 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108205 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108214 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108224 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f4286a-6f10-46bc-960b-ae6b3178f744-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108232 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108241 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.108361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.108250 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f4286a-6f10-46bc-960b-ae6b3178f744-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:04:49.364068 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.364029 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp"] Oct 30 14:04:49.372283 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.372247 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-310546-openshift-default-6cf5648c8f-thprp"] Oct 30 14:04:49.890675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.890635 5693 patch_prober.go:29] interesting pod/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Oct 30 14:04:49.891193 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:49.890717 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Oct 30 14:04:50.261002 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:50.260966 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" path="/var/lib/kubelet/pods/75f4286a-6f10-46bc-960b-ae6b3178f744/volumes" Oct 30 14:04:50.605648 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:50.605375 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:50.739516 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:50.739482 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:50.895270 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:50.895194 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:04:50.895735 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:50.895472 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:04:50.896232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:50.896211 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:04:52.233841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:52.233811 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:53.056525 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:04:53.056480 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:04:53.923577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:53.923542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:54.738187 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:54.738144 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:54.891810 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:54.891768 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:56.603124 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:56.603078 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:57.312968 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:57.312931 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:58.685822 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:58.685785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:58.910786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:04:58.910696 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:00.541425 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:00.541376 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:02.218836 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:02.218801 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:03.906145 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:03.906103 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:05.155333 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.155298 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw"] Oct 30 14:05:05.155815 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.155546 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" podUID="d00101a5-da7d-4c82-a16d-67ca567c9d81" containerName="maas-api" containerID="cri-o://ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646" gracePeriod=30 Oct 30 14:05:05.259896 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.259855 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:05.473345 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.473316 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 14:05:05.652649 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.652565 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls\") pod \"d00101a5-da7d-4c82-a16d-67ca567c9d81\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " Oct 30 14:05:05.653177 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.653150 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca\") pod \"d00101a5-da7d-4c82-a16d-67ca567c9d81\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " Oct 30 14:05:05.653529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.653493 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c\" (UniqueName: \"kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c\") pod \"d00101a5-da7d-4c82-a16d-67ca567c9d81\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " Oct 30 14:05:05.653792 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.653772 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs\") pod \"d00101a5-da7d-4c82-a16d-67ca567c9d81\" (UID: \"d00101a5-da7d-4c82-a16d-67ca567c9d81\") " Oct 30 14:05:05.655642 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.655611 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca" pod "d00101a5-da7d-4c82-a16d-67ca567c9d81" (UID: "d00101a5-da7d-4c82-a16d-67ca567c9d81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:05:05.676965 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.675830 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls" pod "d00101a5-da7d-4c82-a16d-67ca567c9d81" (UID: "d00101a5-da7d-4c82-a16d-67ca567c9d81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:05:05.677589 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.677557 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c" pod "d00101a5-da7d-4c82-a16d-67ca567c9d81" (UID: "d00101a5-da7d-4c82-a16d-67ca567c9d81"). InnerVolumeSpecName "kube-api-access-zt86c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:05:05.679641 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.679615 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs" pod "d00101a5-da7d-4c82-a16d-67ca567c9d81" (UID: "d00101a5-da7d-4c82-a16d-67ca567c9d81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:05:05.756400 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.756362 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:05:05.756400 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.756400 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00101a5-da7d-4c82-a16d-67ca567c9d81-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:05:05.756789 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.756415 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt86c\" (UniqueName: \"kubernetes.io/projected/d00101a5-da7d-4c82-a16d-67ca567c9d81-kube-api-access-zt86c\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:05:05.756789 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.756429 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00101a5-da7d-4c82-a16d-67ca567c9d81-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:05:05.892612 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:05.892567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:06.061284 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.061237 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:06.122800 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.122762 5693 generic.go:356] "Generic (PLEG): container finished" podID="d00101a5-da7d-4c82-a16d-67ca567c9d81" containerID="ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646" exitCode=0 Oct 30 14:05:06.122977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.122809 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" Oct 30 14:05:06.122977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.122842 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" event={"ID":"d00101a5-da7d-4c82-a16d-67ca567c9d81","Type":"ContainerDied","Data":"ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646"} Oct 30 14:05:06.122977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.122904 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw" event={"ID":"d00101a5-da7d-4c82-a16d-67ca567c9d81","Type":"ContainerDied","Data":"09cb42242a2b61372349db83ecad6391c32011a8f52fc006d449ae1550420da0"} Oct 30 14:05:06.122977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.122931 5693 scope.go:122] "RemoveContainer" containerID="ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646" Oct 30 14:05:06.136361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.136342 5693 scope.go:122] "RemoveContainer" containerID="ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646" Oct 30 14:05:06.136659 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:05:06.136635 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646\": container with ID starting with ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646 not found: ID does not exist" containerID="ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646" Oct 30 14:05:06.136772 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.136667 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646"} err="failed to get container status \"ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646\": rpc error: code = NotFound desc = could not find container \"ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646\": container with ID starting with ecdf5d9ba0d1b095799c70e1dea3be4fe55aa9e0e4bc0a0511f177cec4b07646 not found: ID does not exist" Oct 30 14:05:06.174831 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.174798 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw"] Oct 30 14:05:06.186223 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.186197 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca0243-8c9fcc49d-hxdkw"] Oct 30 14:05:06.256464 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.256436 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00101a5-da7d-4c82-a16d-67ca567c9d81" path="/var/lib/kubelet/pods/d00101a5-da7d-4c82-a16d-67ca567c9d81/volumes" Oct 30 14:05:06.584111 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.584077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:06.928481 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:06.928372 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:07.760153 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:07.760121 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:08.039913 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:08.039822 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:08.417063 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:08.416957 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:09.700919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:09.700884 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:11.577589 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:11.577549 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:13.192714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:13.192672 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:14.356496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:14.356462 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:14.543942 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:14.543895 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:14.745056 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:14.745025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:15.214665 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:15.214532 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:16.491246 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:16.491212 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:16.934719 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:16.934625 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:18.646058 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:18.646016 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:20.404115 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:20.404077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:22.139570 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:22.139533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:23.055805 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:05:23.055764 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:05:23.815225 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:23.815188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:25.379840 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:25.379756 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:26.846227 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:26.846191 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:26.986746 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:26.986708 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:27.195634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:27.195529 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:28.804607 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:28.804557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:30.432917 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:30.432878 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:32.026774 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:32.026724 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:33.549884 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:33.549849 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:34.184707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:34.184672 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:34.335005 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:34.334968 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:34.720945 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:34.720913 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:36.126264 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:36.126225 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:37.745306 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:37.745196 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:39.347032 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:39.346984 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:40.925425 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:40.925390 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:41.211541 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:41.211450 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:41.354549 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:41.354516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:41.545733 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:41.545695 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:42.680034 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:42.679987 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:44.472588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:44.472557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:46.062833 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:46.062799 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:47.722117 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:47.722081 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:49.392510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:49.392467 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:51.008051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:51.008019 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:52.608615 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:52.608575 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:52.744813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:52.744775 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:53.083520 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:05:53.083474 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:05:54.227774 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:54.227734 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:55.813787 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:55.813743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:57.491021 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:57.490988 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:58.804164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:58.804133 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:59.182150 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:05:59.182053 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:00.812476 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:00.812441 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:02.497107 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:02.497069 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:04.128810 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:04.128751 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:04.868543 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:04.868503 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:05.852912 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:05.852874 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:07.521034 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:07.520994 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:09.256063 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:09.256022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:10.896947 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:10.896914 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:12.627612 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:12.627558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:12.773783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:12.773743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:14.208191 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:14.208148 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:15.800936 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.800893 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:15.836769 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.836733 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g"] Oct 30 14:06:15.837496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837478 5693 cpu_manager.go:414] "containerMap: removing container" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" Oct 30 14:06:15.837496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837497 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" Oct 30 14:06:15.837661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837535 5693 cpu_manager.go:414] "containerMap: removing container" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" Oct 30 14:06:15.837661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837541 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" Oct 30 14:06:15.837661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837555 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d00101a5-da7d-4c82-a16d-67ca567c9d81" containerName="maas-api" Oct 30 14:06:15.837661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837561 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00101a5-da7d-4c82-a16d-67ca567c9d81" containerName="maas-api" Oct 30 14:06:15.837819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837712 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00101a5-da7d-4c82-a16d-67ca567c9d81" containerName="maas-api" Oct 30 14:06:15.837819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.837740 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" Oct 30 14:06:15.843104 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.843078 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.861826 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.861786 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g"] Oct 30 14:06:15.868148 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.868109 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3f24fd\"/\"openshift-service-ca.crt\"" Oct 30 14:06:15.868734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.868353 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3f24fd\"/\"kube-root-ca.crt\"" Oct 30 14:06:15.869207 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.869171 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3f24fd\"/\"default-dockercfg-zf6kb\"" Oct 30 14:06:15.869207 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.869175 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3f24fd\"/\"rate-test-model-3f24fd-kserve-self-signed-certs\"" Oct 30 14:06:15.938983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.938944 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dhj\" (UniqueName: \"kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.939182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.939110 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.939182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.939171 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.939295 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.939268 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.939362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.939304 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.939422 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.939382 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:15.947220 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:15.947185 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:16.040993 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.040954 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041186 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041100 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041186 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041129 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041186 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041147 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041374 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041184 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041444 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25dhj\" (UniqueName: \"kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041455 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041569 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041546 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.041691 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.041667 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.043510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.043486 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.043829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.043811 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.060243 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.060176 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dhj\" (UniqueName: \"kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj\") pod \"rate-test-model-3f24fd-kserve-6bd68b8c49-b827g\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.158310 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.158276 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:16.476882 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.476847 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g"] Oct 30 14:06:16.478583 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:06:16.478558 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7014e751_7cca_4fe1_bfc4_99170da7256a.slice/crio-2abcfb7a0343de5ccd8e4fc218d7f80c6061ec4b1c22baffe46fcf42d60a7c48 WatchSource:0}: Error finding container 2abcfb7a0343de5ccd8e4fc218d7f80c6061ec4b1c22baffe46fcf42d60a7c48: Status 404 returned error can't find the container with id 2abcfb7a0343de5ccd8e4fc218d7f80c6061ec4b1c22baffe46fcf42d60a7c48 Oct 30 14:06:16.714824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.705790 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" event={"ID":"7014e751-7cca-4fe1-bfc4-99170da7256a","Type":"ContainerStarted","Data":"4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e"} Oct 30 14:06:16.714824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.705876 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" event={"ID":"7014e751-7cca-4fe1-bfc4-99170da7256a","Type":"ContainerStarted","Data":"2abcfb7a0343de5ccd8e4fc218d7f80c6061ec4b1c22baffe46fcf42d60a7c48"} Oct 30 14:06:16.748450 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:16.748345 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" podStartSLOduration=1.748331423 podStartE2EDuration="1.748331423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:16.744770608 +0000 UTC m=+3752.611388781" watchObservedRunningTime="2026-10-30 14:06:16.748331423 +0000 UTC m=+3752.614949595" Oct 30 14:06:17.158783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:17.158701 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:17.513912 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:17.513879 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:19.332771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:19.332722 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:19.937128 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:19.937082 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:20.869010 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:20.868973 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:22.421096 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:22.421054 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:23.056474 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:06:23.056435 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:06:24.084033 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:24.083991 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:25.699621 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:25.699568 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:26.158919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:26.158878 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:26.171697 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:26.171664 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:26.792714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:26.792682 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:06:27.438189 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:27.438094 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:28.886342 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:28.886305 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:29.058420 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:29.058383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:29.228284 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:29.228249 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:30.743657 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:30.743579 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:31.219406 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:31.219324 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:31.358582 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:31.358548 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:31.840835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:31.840801 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:33.037843 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:33.037794 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:34.773954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:34.773913 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:36.519265 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:36.519226 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:38.210781 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:38.210739 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:38.970294 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:38.970258 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:39.212291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:39.212258 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:39.397362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:39.397324 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:40.012460 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:40.012424 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:41.526448 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:41.526416 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:42.221273 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:42.221227 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:42.374897 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:42.374843 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:42.778719 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:42.778679 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:42.973057 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:42.973022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:44.220699 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:44.220665 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:45.915745 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:45.915705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:47.778469 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:47.778400 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:49.438563 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:49.438534 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:49.588252 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:49.588212 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:49.872222 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:49.872183 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:50.059165 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:50.059135 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:50.100719 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:50.100687 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn"] Oct 30 14:06:50.239085 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:50.239052 5693 scope.go:122] "RemoveContainer" containerID="8107aef006e2b078042a9695826ac97f94755c21dce3126a2dc20dac71267d56" Oct 30 14:06:50.239642 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:50.239574 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7"] Oct 30 14:06:50.377891 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:50.377855 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:50.565851 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:50.565763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:51.400456 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:51.400369 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:53.057486 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:06:53.057445 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:06:53.243851 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:53.243818 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:54.806860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:54.806817 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:55.101357 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:55.101236 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerName="payload-processing" containerID="cri-o://b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626" gracePeriod=25 Oct 30 14:06:55.240629 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:55.240563 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerName="payload-pre-processing" containerID="cri-o://08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa" gracePeriod=25 Oct 30 14:06:55.751584 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:55.751548 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:06:56.393640 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:56.393584 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:56.767702 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:56.767654 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:06:58.043266 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:58.043229 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:58.913645 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:58.913607 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:59.615334 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:06:59.615292 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:01.205047 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:01.205014 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:01.364164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:01.364120 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:01.507528 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:01.507495 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:03.098405 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:03.098355 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:04.639707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:04.639671 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:05.751543 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:05.751499 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:07:06.154182 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:06.154088 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:06.767918 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:06.767880 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:07:07.823647 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:07.823533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:09.061519 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.061487 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj"] Oct 30 14:07:09.062148 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.061769 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" podUID="74069549-7659-4791-8965-836da15e510c" containerName="maas-api" containerID="cri-o://650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2" gracePeriod=30 Oct 30 14:07:09.160515 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.160477 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:09.362751 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.362719 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:07:09.398291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.398257 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq\" (UniqueName: \"kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq\") pod \"74069549-7659-4791-8965-836da15e510c\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " Oct 30 14:07:09.398504 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.398323 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs\") pod \"74069549-7659-4791-8965-836da15e510c\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " Oct 30 14:07:09.398504 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.398418 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls\") pod \"74069549-7659-4791-8965-836da15e510c\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " Oct 30 14:07:09.398687 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.398666 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca\") pod \"74069549-7659-4791-8965-836da15e510c\" (UID: \"74069549-7659-4791-8965-836da15e510c\") " Oct 30 14:07:09.399105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.399049 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca" pod "74069549-7659-4791-8965-836da15e510c" (UID: "74069549-7659-4791-8965-836da15e510c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:09.399471 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.399437 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74069549-7659-4791-8965-836da15e510c-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.401378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.401338 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs" pod "74069549-7659-4791-8965-836da15e510c" (UID: "74069549-7659-4791-8965-836da15e510c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:09.401499 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.401377 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls" pod "74069549-7659-4791-8965-836da15e510c" (UID: "74069549-7659-4791-8965-836da15e510c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:09.401917 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.401887 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq" pod "74069549-7659-4791-8965-836da15e510c" (UID: "74069549-7659-4791-8965-836da15e510c"). InnerVolumeSpecName "kube-api-access-g27sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:09.500480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.500436 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27sq\" (UniqueName: \"kubernetes.io/projected/74069549-7659-4791-8965-836da15e510c-kube-api-access-g27sq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.500480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.500475 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.500480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.500485 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74069549-7659-4791-8965-836da15e510c-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.773616 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.773563 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:09.919450 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:09.919387 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:10.172778 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.172693 5693 generic.go:356] "Generic (PLEG): container finished" podID="74069549-7659-4791-8965-836da15e510c" containerID="650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2" exitCode=0 Oct 30 14:07:10.172778 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.172739 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" Oct 30 14:07:10.173232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.172778 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" event={"ID":"74069549-7659-4791-8965-836da15e510c","Type":"ContainerDied","Data":"650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2"} Oct 30 14:07:10.173232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.172829 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj" event={"ID":"74069549-7659-4791-8965-836da15e510c","Type":"ContainerDied","Data":"a1e60990c954c378e7d7239f0374615bfef319e6048d5267a9b81345820aa2fa"} Oct 30 14:07:10.173232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.172851 5693 scope.go:122] "RemoveContainer" containerID="650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2" Oct 30 14:07:10.185336 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.185318 5693 scope.go:122] "RemoveContainer" containerID="650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2" Oct 30 14:07:10.185630 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:10.185593 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2\": container with ID starting with 650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2 not found: ID does not exist" containerID="650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2" Oct 30 14:07:10.185717 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.185640 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2"} err="failed to get container status \"650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2\": rpc error: code = NotFound desc = could not find container \"650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2\": container with ID starting with 650d381f743203e111c919c82d4ef3336a3a30d22fa6252108295bbf5a1ef3e2 not found: ID does not exist" Oct 30 14:07:10.223480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.223442 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj"] Oct 30 14:07:10.235222 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.235182 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b32c32a-5c8446f88f-pcxgj"] Oct 30 14:07:10.256825 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.256800 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74069549-7659-4791-8965-836da15e510c" path="/var/lib/kubelet/pods/74069549-7659-4791-8965-836da15e510c/volumes" Oct 30 14:07:10.338420 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:10.338390 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:11.530928 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:11.530885 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:13.106295 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:13.106260 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:14.714071 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:14.714029 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:15.392221 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.392194 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 14:07:15.499653 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.499625 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r\" (UniqueName: \"kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r\") pod \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " Oct 30 14:07:15.499825 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.499742 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume\") pod \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\" (UID: \"0a45b9f7-0e77-4385-a4f5-a0193805c4fb\") " Oct 30 14:07:15.500180 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.500151 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume" pod "0a45b9f7-0e77-4385-a4f5-a0193805c4fb" (UID: "0a45b9f7-0e77-4385-a4f5-a0193805c4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:15.500405 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.500390 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:15.501886 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.501860 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r" pod "0a45b9f7-0e77-4385-a4f5-a0193805c4fb" (UID: "0a45b9f7-0e77-4385-a4f5-a0193805c4fb"). InnerVolumeSpecName "kube-api-access-6n68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:15.507563 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.507541 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 14:07:15.601584 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.601547 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume\") pod \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " Oct 30 14:07:15.601799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.601761 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd\" (UniqueName: \"kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd\") pod \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\" (UID: \"31b4bc3e-9381-47bc-b53a-608d6bbd15f4\") " Oct 30 14:07:15.601937 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.601911 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume" pod "31b4bc3e-9381-47bc-b53a-608d6bbd15f4" (UID: "31b4bc3e-9381-47bc-b53a-608d6bbd15f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:15.602400 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.602379 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:15.602495 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.602401 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n68r\" (UniqueName: \"kubernetes.io/projected/0a45b9f7-0e77-4385-a4f5-a0193805c4fb-kube-api-access-6n68r\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:15.603873 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.603850 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd" pod "31b4bc3e-9381-47bc-b53a-608d6bbd15f4" (UID: "31b4bc3e-9381-47bc-b53a-608d6bbd15f4"). InnerVolumeSpecName "kube-api-access-42dbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:15.703792 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:15.703704 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dbd\" (UniqueName: \"kubernetes.io/projected/31b4bc3e-9381-47bc-b53a-608d6bbd15f4-kube-api-access-42dbd\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:16.223104 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.223069 5693 generic.go:356] "Generic (PLEG): container finished" podID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerID="b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626" exitCode=0 Oct 30 14:07:16.223633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.223120 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" Oct 30 14:07:16.223633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.223152 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" event={"ID":"0a45b9f7-0e77-4385-a4f5-a0193805c4fb","Type":"ContainerDied","Data":"b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626"} Oct 30 14:07:16.223633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.223214 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn" event={"ID":"0a45b9f7-0e77-4385-a4f5-a0193805c4fb","Type":"ContainerDied","Data":"0ee387512ad453b5faa1102af7b345eee05511aff71bc1a500b22e63ab66a5b2"} Oct 30 14:07:16.223633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.223241 5693 scope.go:122] "RemoveContainer" containerID="b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626" Oct 30 14:07:16.225179 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.225150 5693 generic.go:356] "Generic (PLEG): container finished" podID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerID="08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa" exitCode=0 Oct 30 14:07:16.225314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.225201 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" event={"ID":"31b4bc3e-9381-47bc-b53a-608d6bbd15f4","Type":"ContainerDied","Data":"08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa"} Oct 30 14:07:16.225314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.225248 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" event={"ID":"31b4bc3e-9381-47bc-b53a-608d6bbd15f4","Type":"ContainerDied","Data":"6ac2fc4d2eb06c92ecca58a243dde1b7d2e1925d64cb5e7b693f974c640e290c"} Oct 30 14:07:16.225314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.225207 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7" Oct 30 14:07:16.238110 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.238057 5693 scope.go:122] "RemoveContainer" containerID="b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626" Oct 30 14:07:16.238451 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:16.238424 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626\": container with ID starting with b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626 not found: ID does not exist" containerID="b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626" Oct 30 14:07:16.238633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.238462 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626"} err="failed to get container status \"b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626\": rpc error: code = NotFound desc = could not find container \"b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626\": container with ID starting with b57177c5f2542cd2a820ab248dd5131863e9dfd7df5f1faaf2596ae262353626 not found: ID does not exist" Oct 30 14:07:16.238633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.238487 5693 scope.go:122] "RemoveContainer" containerID="08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa" Oct 30 14:07:16.249106 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.249082 5693 scope.go:122] "RemoveContainer" containerID="08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa" Oct 30 14:07:16.249445 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:16.249426 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa\": container with ID starting with 08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa not found: ID does not exist" containerID="08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa" Oct 30 14:07:16.249530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.249453 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa"} err="failed to get container status \"08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa\": rpc error: code = NotFound desc = could not find container \"08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa\": container with ID starting with 08da1de841378f796249f4adad7fa445d10628d7e05181e593f57a916a4d76aa not found: ID does not exist" Oct 30 14:07:16.288485 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.288446 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7"] Oct 30 14:07:16.291665 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.291638 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-552cc3-645bdff47-ljtd7"] Oct 30 14:07:16.320164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.320132 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn"] Oct 30 14:07:16.324061 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.324032 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-552cc3-7c8549b4cd-s54zn"] Oct 30 14:07:16.334959 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.334923 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:16.970365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:16.970334 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:17.166248 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:17.166212 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:17.285234 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:17.285203 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs"] Oct 30 14:07:17.285869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:17.285839 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" containerID="cri-o://5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333" gracePeriod=30 Oct 30 14:07:17.344064 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:17.344031 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:18.126960 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:18.126928 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:18.255534 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:18.255499 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" path="/var/lib/kubelet/pods/0a45b9f7-0e77-4385-a4f5-a0193805c4fb/volumes" Oct 30 14:07:18.256010 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:18.255994 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" path="/var/lib/kubelet/pods/31b4bc3e-9381-47bc-b53a-608d6bbd15f4/volumes" Oct 30 14:07:19.117240 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.117180 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:19.461033 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.460946 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7"] Oct 30 14:07:19.461556 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.461506 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" podUID="5dedcc23-b119-473d-8c63-cfe2ec549680" containerName="maas-api" containerID="cri-o://7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6" gracePeriod=30 Oct 30 14:07:19.581143 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.581112 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:19.791485 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.791454 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 14:07:19.971024 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.970985 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs\" (UniqueName: \"kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs\") pod \"5dedcc23-b119-473d-8c63-cfe2ec549680\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " Oct 30 14:07:19.971226 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.971127 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls\") pod \"5dedcc23-b119-473d-8c63-cfe2ec549680\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " Oct 30 14:07:19.971226 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.971191 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs\") pod \"5dedcc23-b119-473d-8c63-cfe2ec549680\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " Oct 30 14:07:19.971369 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.971248 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca\") pod \"5dedcc23-b119-473d-8c63-cfe2ec549680\" (UID: \"5dedcc23-b119-473d-8c63-cfe2ec549680\") " Oct 30 14:07:19.971800 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.971764 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca" pod "5dedcc23-b119-473d-8c63-cfe2ec549680" (UID: "5dedcc23-b119-473d-8c63-cfe2ec549680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:19.971977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.971960 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dedcc23-b119-473d-8c63-cfe2ec549680-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:19.973528 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.973491 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs" pod "5dedcc23-b119-473d-8c63-cfe2ec549680" (UID: "5dedcc23-b119-473d-8c63-cfe2ec549680"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:19.973528 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.973515 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls" pod "5dedcc23-b119-473d-8c63-cfe2ec549680" (UID: "5dedcc23-b119-473d-8c63-cfe2ec549680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:19.973729 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:19.973546 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs" pod "5dedcc23-b119-473d-8c63-cfe2ec549680" (UID: "5dedcc23-b119-473d-8c63-cfe2ec549680"). InnerVolumeSpecName "kube-api-access-vd6cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:20.073371 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.073274 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6cs\" (UniqueName: \"kubernetes.io/projected/5dedcc23-b119-473d-8c63-cfe2ec549680-kube-api-access-vd6cs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:20.073371 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.073316 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:20.073371 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.073329 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dedcc23-b119-473d-8c63-cfe2ec549680-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:20.158257 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.158215 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:20.311571 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.311536 5693 generic.go:356] "Generic (PLEG): container finished" podID="5dedcc23-b119-473d-8c63-cfe2ec549680" containerID="7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6" exitCode=0 Oct 30 14:07:20.311777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.311591 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" Oct 30 14:07:20.311777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.311630 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" event={"ID":"5dedcc23-b119-473d-8c63-cfe2ec549680","Type":"ContainerDied","Data":"7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6"} Oct 30 14:07:20.311777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.311673 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7" event={"ID":"5dedcc23-b119-473d-8c63-cfe2ec549680","Type":"ContainerDied","Data":"323bf6f671567efceade0f19fdb9d0a8b7963b5f2d0105c2253e3d892f2a39f0"} Oct 30 14:07:20.311777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.311702 5693 scope.go:122] "RemoveContainer" containerID="7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6" Oct 30 14:07:20.324734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.324705 5693 scope.go:122] "RemoveContainer" containerID="7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6" Oct 30 14:07:20.325122 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:20.325088 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6\": container with ID starting with 7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6 not found: ID does not exist" containerID="7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6" Oct 30 14:07:20.325273 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.325124 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6"} err="failed to get container status \"7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6\": rpc error: code = NotFound desc = could not find container \"7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6\": container with ID starting with 7b3240b95418f8c7d89ef39b5c4f1e84aee1780bf765504d7d87be87287d01a6 not found: ID does not exist" Oct 30 14:07:20.384917 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.384873 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7"] Oct 30 14:07:20.396700 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.396667 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-552cc3-74dd68c8c9-wnhq7"] Oct 30 14:07:20.443072 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.443035 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:20.787699 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.787668 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:20.898207 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.898166 5693 patch_prober.go:29] interesting pod/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:07:20.898406 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:20.898247 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:07:22.111791 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.111687 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:22.261620 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.258450 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dedcc23-b119-473d-8c63-cfe2ec549680" path="/var/lib/kubelet/pods/5dedcc23-b119-473d-8c63-cfe2ec549680/volumes" Oct 30 14:07:22.579657 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.579629 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:07:22.631326 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631286 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631375 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631396 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631426 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631468 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631672 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631770 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.631983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631835 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:22.632053 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631931 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:22.632053 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631999 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:22.632053 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:22.632053 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.631808 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.632283 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket\") pod \"5b25af17-7179-427d-b5f8-df3fd9f434b7\" (UID: \"5b25af17-7179-427d-b5f8-df3fd9f434b7\") " Oct 30 14:07:22.632440 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632417 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:22.632980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632958 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.632980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632979 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b25af17-7179-427d-b5f8-df3fd9f434b7-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.633170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632989 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.633170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.632998 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.633170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.633006 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.633978 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.633955 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "kube-api-access-55jpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:22.634065 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.633973 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:22.634116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.634069 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:07:22.634282 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.634261 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token" pod "5b25af17-7179-427d-b5f8-df3fd9f434b7" (UID: "5b25af17-7179-427d-b5f8-df3fd9f434b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:22.734838 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.734799 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.734838 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.734833 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.734838 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.734843 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jpq\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-kube-api-access-55jpq\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:22.735159 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:22.734855 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b25af17-7179-427d-b5f8-df3fd9f434b7-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:23.061753 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:23.061660 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:07:23.335876 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.335787 5693 generic.go:356] "Generic (PLEG): container finished" podID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerID="5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333" exitCode=0 Oct 30 14:07:23.335876 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.335842 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" Oct 30 14:07:23.336314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.335877 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" event={"ID":"5b25af17-7179-427d-b5f8-df3fd9f434b7","Type":"ContainerDied","Data":"5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333"} Oct 30 14:07:23.336314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.335936 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs" event={"ID":"5b25af17-7179-427d-b5f8-df3fd9f434b7","Type":"ContainerDied","Data":"bc00e3c15a1f83d52baa54f4af5c721124ae04fca131558f656cb493b01b00a3"} Oct 30 14:07:23.336314 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.335957 5693 scope.go:122] "RemoveContainer" containerID="5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333" Oct 30 14:07:23.348005 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.347984 5693 scope.go:122] "RemoveContainer" containerID="ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a" Oct 30 14:07:23.358142 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.358122 5693 scope.go:122] "RemoveContainer" containerID="5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333" Oct 30 14:07:23.358412 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:23.358391 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333\": container with ID starting with 5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333 not found: ID does not exist" containerID="5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333" Oct 30 14:07:23.358456 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.358420 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333"} err="failed to get container status \"5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333\": rpc error: code = NotFound desc = could not find container \"5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333\": container with ID starting with 5d4ec9f63a607909edc757991264b2c64edc15ffca1fa17ceb633e4b31394333 not found: ID does not exist" Oct 30 14:07:23.358456 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.358439 5693 scope.go:122] "RemoveContainer" containerID="ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a" Oct 30 14:07:23.358703 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:23.358686 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a\": container with ID starting with ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a not found: ID does not exist" containerID="ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a" Oct 30 14:07:23.358763 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.358708 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a"} err="failed to get container status \"ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a\": rpc error: code = NotFound desc = could not find container \"ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a\": container with ID starting with ded052afab116fd44decc950163b91966a61f4fecaecd23ec865e5b0dc29899a not found: ID does not exist" Oct 30 14:07:23.399995 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.399965 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs"] Oct 30 14:07:23.411709 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.411678 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9b32c32a-openshift-default-64cf5c8d85-br2fs"] Oct 30 14:07:23.826026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:23.825856 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:24.264432 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:24.262336 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" path="/var/lib/kubelet/pods/5b25af17-7179-427d-b5f8-df3fd9f434b7/volumes" Oct 30 14:07:25.615590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:25.615556 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:26.921639 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:26.921590 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5"] Oct 30 14:07:26.922445 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:26.922344 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" podUID="73fca334-9e80-4517-a6be-dfe5c436d50b" containerName="maas-api" containerID="cri-o://1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634" gracePeriod=30 Oct 30 14:07:27.023615 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.023567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:27.220300 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.220268 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 14:07:27.311070 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.311034 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca\") pod \"73fca334-9e80-4517-a6be-dfe5c436d50b\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " Oct 30 14:07:27.311294 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.311273 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls\") pod \"73fca334-9e80-4517-a6be-dfe5c436d50b\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " Oct 30 14:07:27.311356 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.311330 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv\" (UniqueName: \"kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv\") pod \"73fca334-9e80-4517-a6be-dfe5c436d50b\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " Oct 30 14:07:27.311465 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.311450 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs\") pod \"73fca334-9e80-4517-a6be-dfe5c436d50b\" (UID: \"73fca334-9e80-4517-a6be-dfe5c436d50b\") " Oct 30 14:07:27.315193 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.315153 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca" pod "73fca334-9e80-4517-a6be-dfe5c436d50b" (UID: "73fca334-9e80-4517-a6be-dfe5c436d50b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:27.318797 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.318240 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv" pod "73fca334-9e80-4517-a6be-dfe5c436d50b" (UID: "73fca334-9e80-4517-a6be-dfe5c436d50b"). InnerVolumeSpecName "kube-api-access-5thvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:27.319028 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.318999 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs" pod "73fca334-9e80-4517-a6be-dfe5c436d50b" (UID: "73fca334-9e80-4517-a6be-dfe5c436d50b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:27.319964 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.319933 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls" pod "73fca334-9e80-4517-a6be-dfe5c436d50b" (UID: "73fca334-9e80-4517-a6be-dfe5c436d50b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:27.370136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.370096 5693 generic.go:356] "Generic (PLEG): container finished" podID="73fca334-9e80-4517-a6be-dfe5c436d50b" containerID="1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634" exitCode=0 Oct 30 14:07:27.370325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.370145 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" Oct 30 14:07:27.370325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.370179 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" event={"ID":"73fca334-9e80-4517-a6be-dfe5c436d50b","Type":"ContainerDied","Data":"1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634"} Oct 30 14:07:27.370325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.370240 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5" event={"ID":"73fca334-9e80-4517-a6be-dfe5c436d50b","Type":"ContainerDied","Data":"7a793fb921b9b8e14b70db2428c1d536c08a91b80de35debc7c57ba4b695578e"} Oct 30 14:07:27.370325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.370264 5693 scope.go:122] "RemoveContainer" containerID="1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634" Oct 30 14:07:27.381976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.381957 5693 scope.go:122] "RemoveContainer" containerID="1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634" Oct 30 14:07:27.382241 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:27.382223 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634\": container with ID starting with 1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634 not found: ID does not exist" containerID="1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634" Oct 30 14:07:27.382292 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.382248 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634"} err="failed to get container status \"1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634\": rpc error: code = NotFound desc = could not find container \"1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634\": container with ID starting with 1aebd311b90fcb0e99d0e7075650202ebf44551a2348d769b76d43d0665ba634 not found: ID does not exist" Oct 30 14:07:27.435701 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.428728 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.435701 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.428766 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thvv\" (UniqueName: \"kubernetes.io/projected/73fca334-9e80-4517-a6be-dfe5c436d50b-kube-api-access-5thvv\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.435701 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.428781 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fca334-9e80-4517-a6be-dfe5c436d50b-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.435701 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.428796 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fca334-9e80-4517-a6be-dfe5c436d50b-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.463622 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.458729 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5"] Oct 30 14:07:27.463824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.463677 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2cecb-5d56cbf4f8-rh9h5"] Oct 30 14:07:27.514391 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.514360 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:27.663046 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.663008 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:27.780914 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.780883 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js"] Oct 30 14:07:27.781122 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.781100 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" containerID="cri-o://3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d" gracePeriod=30 Oct 30 14:07:27.810524 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.810490 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:27.969888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:27.965051 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:28.237256 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:28.237222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:28.255247 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:28.255211 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fca334-9e80-4517-a6be-dfe5c436d50b" path="/var/lib/kubelet/pods/73fca334-9e80-4517-a6be-dfe5c436d50b/volumes" Oct 30 14:07:28.383666 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:28.383567 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:29.537585 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:29.537550 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:31.171324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:31.171275 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:32.720352 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:32.720291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:33.057713 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.057682 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 14:07:33.115520 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115488 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.115749 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115532 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.115749 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115567 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.115749 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115705 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115752 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115802 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115857 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115890 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115939 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.115983 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data\") pod \"34a928a3-4554-425d-a2e5-38a118e6729e\" (UID: \"34a928a3-4554-425d-a2e5-38a118e6729e\") " Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116067 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116141 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116311 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116542 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116863 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116886 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a928a3-4554-425d-a2e5-38a118e6729e-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116900 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116914 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.116929 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.118845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.118681 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "kube-api-access-mjfp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:33.119495 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.118862 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:33.119495 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.119009 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:07:33.119495 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.119133 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token" pod "34a928a3-4554-425d-a2e5-38a118e6729e" (UID: "34a928a3-4554-425d-a2e5-38a118e6729e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:33.218038 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.218004 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a928a3-4554-425d-a2e5-38a118e6729e-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.218038 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.218035 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a928a3-4554-425d-a2e5-38a118e6729e-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.218038 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.218045 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjfp9\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-kube-api-access-mjfp9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.218269 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.218055 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a928a3-4554-425d-a2e5-38a118e6729e-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:07:33.415738 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.415648 5693 generic.go:356] "Generic (PLEG): container finished" podID="34a928a3-4554-425d-a2e5-38a118e6729e" containerID="3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d" exitCode=0 Oct 30 14:07:33.415738 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.415695 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" Oct 30 14:07:33.415950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.415734 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" event={"ID":"34a928a3-4554-425d-a2e5-38a118e6729e","Type":"ContainerDied","Data":"3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d"} Oct 30 14:07:33.415950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.415787 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js" event={"ID":"34a928a3-4554-425d-a2e5-38a118e6729e","Type":"ContainerDied","Data":"4d0f64299c3cfc39647507ae61adf21b13bdcc4a38445059ba39d025c43a2340"} Oct 30 14:07:33.415950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.415809 5693 scope.go:122] "RemoveContainer" containerID="3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d" Oct 30 14:07:33.427752 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.427728 5693 scope.go:122] "RemoveContainer" containerID="0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591" Oct 30 14:07:33.429529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.429505 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:33.437613 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.437578 5693 scope.go:122] "RemoveContainer" containerID="3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d" Oct 30 14:07:33.437893 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:33.437874 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d\": container with ID starting with 3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d not found: ID does not exist" containerID="3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d" Oct 30 14:07:33.437980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.437907 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d"} err="failed to get container status \"3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d\": rpc error: code = NotFound desc = could not find container \"3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d\": container with ID starting with 3086faa38ad5706e0522f2dce87a75bd15b9449f0b41806f0955c7866f55953d not found: ID does not exist" Oct 30 14:07:33.437980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.437929 5693 scope.go:122] "RemoveContainer" containerID="0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591" Oct 30 14:07:33.438210 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:33.438192 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591\": container with ID starting with 0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591 not found: ID does not exist" containerID="0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591" Oct 30 14:07:33.438251 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.438220 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591"} err="failed to get container status \"0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591\": rpc error: code = NotFound desc = could not find container \"0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591\": container with ID starting with 0b28bde3b88aad65cd4330a27f2cbea9233a6dc2774089ad2c5617251d3fc591 not found: ID does not exist" Oct 30 14:07:33.467709 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.467679 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js"] Oct 30 14:07:33.478787 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:33.478760 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-552cc3-openshift-default-5ff487df49-mp9js"] Oct 30 14:07:34.257552 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:34.257521 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" path="/var/lib/kubelet/pods/34a928a3-4554-425d-a2e5-38a118e6729e/volumes" Oct 30 14:07:34.314435 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:34.314393 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:36.011727 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:36.011685 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:36.219024 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:36.218987 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:36.365153 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:36.365069 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:36.671572 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:36.671468 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:37.762199 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:37.762168 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:39.094709 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:39.094673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:39.239437 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:39.239407 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:39.398572 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:39.398475 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:41.011824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:41.011788 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:42.581510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:42.581471 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:44.320685 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:44.320651 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:46.049170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.049135 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:46.257115 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257081 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf"] Oct 30 14:07:46.257837 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257819 5693 cpu_manager.go:414] "containerMap: removing container" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" Oct 30 14:07:46.257895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257839 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" Oct 30 14:07:46.257895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257854 5693 cpu_manager.go:414] "containerMap: removing container" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerName="payload-pre-processing" Oct 30 14:07:46.257895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257860 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerName="payload-pre-processing" Oct 30 14:07:46.257895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257882 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" Oct 30 14:07:46.257895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257887 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257900 5693 cpu_manager.go:414] "containerMap: removing container" podUID="73fca334-9e80-4517-a6be-dfe5c436d50b" containerName="maas-api" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257905 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fca334-9e80-4517-a6be-dfe5c436d50b" containerName="maas-api" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257912 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74069549-7659-4791-8965-836da15e510c" containerName="maas-api" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257917 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74069549-7659-4791-8965-836da15e510c" containerName="maas-api" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257927 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerName="payload-processing" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257932 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerName="payload-processing" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257947 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5dedcc23-b119-473d-8c63-cfe2ec549680" containerName="maas-api" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.257952 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dedcc23-b119-473d-8c63-cfe2ec549680" containerName="maas-api" Oct 30 14:07:46.258060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258054 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fca334-9e80-4517-a6be-dfe5c436d50b" containerName="maas-api" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258063 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74069549-7659-4791-8965-836da15e510c" containerName="maas-api" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258070 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a45b9f7-0e77-4385-a4f5-a0193805c4fb" containerName="payload-processing" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258076 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258082 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dedcc23-b119-473d-8c63-cfe2ec549680" containerName="maas-api" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258089 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f4286a-6f10-46bc-960b-ae6b3178f744" containerName="istio-proxy" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258097 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" Oct 30 14:07:46.258378 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.258104 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b4bc3e-9381-47bc-b53a-608d6bbd15f4" containerName="payload-pre-processing" Oct 30 14:07:46.263460 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.263430 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.284178 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.279653 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf"] Oct 30 14:07:46.287094 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.287046 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"openshift-service-ca.crt\"" Oct 30 14:07:46.287354 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.287327 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"kube-root-ca.crt\"" Oct 30 14:07:46.290493 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.290470 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"default-dockercfg-7wx22\"" Oct 30 14:07:46.290493 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.290476 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"rate-test-model-29a3f3-kserve-self-signed-certs\"" Oct 30 14:07:46.356927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.356834 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:46.411968 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.411933 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.412149 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.411982 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.412149 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.412074 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.412149 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.412109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cqj\" (UniqueName: \"kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.412261 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.412192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.412427 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.412413 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.513824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.513781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.514026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.513837 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.514026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.513899 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.514026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.513947 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cqj\" (UniqueName: \"kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.514026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.513973 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.514026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.513998 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.514393 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.514374 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.515143 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.514930 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.515143 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.515068 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.536959 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.536915 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.537750 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.537725 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.538166 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.538122 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cqj\" (UniqueName: \"kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj\") pod \"rate-test-model-29a3f3-kserve-768949df97-dq2kf\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.583699 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.583659 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:46.892454 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:46.892426 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf"] Oct 30 14:07:46.895057 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:07:46.895030 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d967b1c_4a62_49ff_bbc0_0e5900fd7d14.slice/crio-fcc4431921a4458e055aa509a6d41ba9050beca0b84745f86aa3e833a97d37d7 WatchSource:0}: Error finding container fcc4431921a4458e055aa509a6d41ba9050beca0b84745f86aa3e833a97d37d7: Status 404 returned error can't find the container with id fcc4431921a4458e055aa509a6d41ba9050beca0b84745f86aa3e833a97d37d7 Oct 30 14:07:47.522396 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:47.522351 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" event={"ID":"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14","Type":"ContainerStarted","Data":"1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f"} Oct 30 14:07:47.522874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:47.522407 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" event={"ID":"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14","Type":"ContainerStarted","Data":"fcc4431921a4458e055aa509a6d41ba9050beca0b84745f86aa3e833a97d37d7"} Oct 30 14:07:47.562611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:47.562550 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" podStartSLOduration=1.562535451 podStartE2EDuration="1.562535451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:47.560791521 +0000 UTC m=+3843.427409694" watchObservedRunningTime="2026-10-30 14:07:47.562535451 +0000 UTC m=+3843.429153623" Oct 30 14:07:47.584326 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:47.584292 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:47.677274 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:47.677241 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:47.813618 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:47.813516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:49.376735 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:49.376700 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:51.004869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:51.004828 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:52.534770 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:52.534736 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:53.055908 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:07:53.055866 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:07:54.488397 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:54.488364 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:55.801957 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:55.801907 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:56.584639 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:56.584592 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:56.598350 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:56.598322 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:57.403566 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:57.403529 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:57.597154 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:57.597120 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:07:58.969796 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:58.969763 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:59.107573 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:07:59.107540 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:00.642627 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:00.642564 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:02.695916 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:02.695877 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:04.314804 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:04.314767 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:05.901757 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:05.901706 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:07.535302 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:07.535254 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:09.219771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:09.219711 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:09.631326 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:09.631293 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:09.797938 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:09.797909 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:10.767955 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:10.767919 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:11.375899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:11.375863 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:11.521894 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:11.521854 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:11.981188 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:11.981155 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:13.192869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:13.192833 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:13.504087 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:13.503588 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:15.046880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:15.046841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:16.659995 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:16.659953 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:18.409206 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:18.409168 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:18.696142 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:18.696059 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:18.847311 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:18.847268 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:19.041944 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:19.041905 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:20.227088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:20.227054 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:21.475083 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:21.475047 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:21.628888 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:21.628844 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:22.083014 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:22.082974 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:23.056920 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:08:23.056871 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:08:23.897987 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:23.897922 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:25.594661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:25.594274 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:27.279305 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:27.279259 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:28.914739 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:28.914704 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:30.130628 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:30.130576 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:30.295012 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:30.294972 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:30.500079 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:30.500042 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:32.109887 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:32.109851 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:33.700313 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:33.700274 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:35.441620 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:35.441517 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:36.775906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:36.775868 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:37.235057 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:37.235022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:37.386840 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:37.386805 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:37.875740 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:37.875687 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:39.023295 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:39.023255 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:40.543653 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:40.542861 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:42.071226 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:42.071194 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:43.685900 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:43.685857 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:44.973010 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:44.972972 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:45.128498 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:45.128465 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:45.286566 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:45.286512 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:45.432328 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:45.432294 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:47.023312 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:47.023264 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:48.617994 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:48.617952 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:50.307712 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:50.307679 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:52.025241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:52.025146 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:53.056071 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:08:53.056027 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:08:53.709722 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:53.709686 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:55.323381 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:55.323341 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:56.652869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:56.652834 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:57.203275 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:57.203239 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:57.362615 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:57.362551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:57.817229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:57.817195 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:58.998999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:58.998965 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:59.150230 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:59.150198 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:59.295302 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:08:59.295210 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:00.620308 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:00.620266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:02.211109 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:02.211071 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:03.844702 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:03.844650 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:04.359555 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:04.359520 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:04.560062 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:04.560020 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:04.640140 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:04.640050 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd"] Oct 30 14:09:04.640338 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:04.640306 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" containerID="cri-o://3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89" gracePeriod=30 Oct 30 14:09:04.714965 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:04.714922 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:05.207953 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:05.207924 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:05.460852 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:05.460756 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:07.262428 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:07.262382 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:08.892846 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:08.892809 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:09.951689 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:09.951659 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 14:09:10.058545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.058504 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.058745 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.058641 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.058745 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.058707 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.058854 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.058837 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.058934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.058919 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.058999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.058974 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.059064 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059008 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.059064 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059008 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:10.059257 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059097 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.059257 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059141 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs\") pod \"d687829a-5d4b-469f-b1a3-262f612892a0\" (UID: \"d687829a-5d4b-469f-b1a3-262f612892a0\") " Oct 30 14:09:10.059375 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059312 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:10.059650 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059547 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:10.059650 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059611 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:10.059880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059841 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.059880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059863 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.059880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059876 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.060068 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059891 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.060068 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.059936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:10.061069 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.061041 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:10.061185 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.061052 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "kube-api-access-khrt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:10.061317 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.061294 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:09:10.061706 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.061690 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token" pod "d687829a-5d4b-469f-b1a3-262f612892a0" (UID: "d687829a-5d4b-469f-b1a3-262f612892a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:10.123468 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.123427 5693 generic.go:356] "Generic (PLEG): container finished" podID="d687829a-5d4b-469f-b1a3-262f612892a0" containerID="3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89" exitCode=0 Oct 30 14:09:10.123661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.123481 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" event={"ID":"d687829a-5d4b-469f-b1a3-262f612892a0","Type":"ContainerDied","Data":"3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89"} Oct 30 14:09:10.123661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.123524 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" event={"ID":"d687829a-5d4b-469f-b1a3-262f612892a0","Type":"ContainerDied","Data":"2ce1e7be15f023a2e73904e45a93f53ea43351139f616409f6170f86bf43ee92"} Oct 30 14:09:10.123661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.123546 5693 scope.go:122] "RemoveContainer" containerID="3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89" Oct 30 14:09:10.123661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.123483 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd" Oct 30 14:09:10.135282 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.135257 5693 scope.go:122] "RemoveContainer" containerID="b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737" Oct 30 14:09:10.144708 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.144690 5693 scope.go:122] "RemoveContainer" containerID="3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89" Oct 30 14:09:10.144960 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:10.144938 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89\": container with ID starting with 3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89 not found: ID does not exist" containerID="3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89" Oct 30 14:09:10.145031 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.144973 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89"} err="failed to get container status \"3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89\": rpc error: code = NotFound desc = could not find container \"3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89\": container with ID starting with 3810c90a254cc39461fa198565cf03cd0f4205514fff9c32df727057dfbe0b89 not found: ID does not exist" Oct 30 14:09:10.145031 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.145000 5693 scope.go:122] "RemoveContainer" containerID="b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737" Oct 30 14:09:10.145234 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:10.145214 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737\": container with ID starting with b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737 not found: ID does not exist" containerID="b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737" Oct 30 14:09:10.145276 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.145241 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737"} err="failed to get container status \"b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737\": rpc error: code = NotFound desc = could not find container \"b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737\": container with ID starting with b748f5ae10c391b3380d0f1f1330577b169a6e61b8786b393adabd6e34374737 not found: ID does not exist" Oct 30 14:09:10.159166 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.159137 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf"] Oct 30 14:09:10.161950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.161923 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687829a-5d4b-469f-b1a3-262f612892a0-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.162092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.161954 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.162092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.161969 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khrt4\" (UniqueName: \"kubernetes.io/projected/d687829a-5d4b-469f-b1a3-262f612892a0-kube-api-access-khrt4\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.162092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.161983 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687829a-5d4b-469f-b1a3-262f612892a0-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.162092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.161998 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687829a-5d4b-469f-b1a3-262f612892a0-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:10.179588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.179510 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd"] Oct 30 14:09:10.187210 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.187185 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f12b80-openshift-default-6748c54cf5-mlxjd"] Oct 30 14:09:10.256885 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.256849 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" path="/var/lib/kubelet/pods/d687829a-5d4b-469f-b1a3-262f612892a0/volumes" Oct 30 14:09:10.310996 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.310962 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:10.509816 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:10.509781 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:12.206933 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:12.206898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:13.919522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:13.919478 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:15.749784 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:15.749750 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:15.923766 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:15.923733 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t"] Oct 30 14:09:15.958689 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:15.958646 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5"] Oct 30 14:09:16.145767 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:16.145684 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:16.346245 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:16.346203 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:17.556689 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:17.556652 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:19.118667 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:19.118629 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:20.802718 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:20.802590 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:20.924142 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:20.924098 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" podUID="3789b872-e323-4b81-a952-6783a7209ece" containerName="payload-processing" containerID="cri-o://d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e" gracePeriod=25 Oct 30 14:09:20.959110 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:20.959076 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerName="payload-pre-processing" containerID="cri-o://0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706" gracePeriod=25 Oct 30 14:09:22.605310 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:22.605261 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:23.056351 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:23.056314 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:09:24.220397 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:24.220293 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:25.182310 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.182273 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" containerID="cri-o://b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed" gracePeriod=45 Oct 30 14:09:25.451113 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.451037 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 14:09:25.483592 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.483559 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home\") pod \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " Oct 30 14:09:25.483795 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.483640 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs\") pod \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " Oct 30 14:09:25.483795 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.483682 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk\" (UniqueName: \"kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk\") pod \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " Oct 30 14:09:25.483932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.483854 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache\") pod \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " Oct 30 14:09:25.483932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.483897 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home" pod "4529cf2d-afc8-4338-b21e-65bdf6cdf76e" (UID: "4529cf2d-afc8-4338-b21e-65bdf6cdf76e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:25.484051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.484016 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm\") pod \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " Oct 30 14:09:25.484200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.484123 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir\") pod \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\" (UID: \"4529cf2d-afc8-4338-b21e-65bdf6cdf76e\") " Oct 30 14:09:25.484200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.484138 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache" pod "4529cf2d-afc8-4338-b21e-65bdf6cdf76e" (UID: "4529cf2d-afc8-4338-b21e-65bdf6cdf76e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:25.484775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.484749 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir" pod "4529cf2d-afc8-4338-b21e-65bdf6cdf76e" (UID: "4529cf2d-afc8-4338-b21e-65bdf6cdf76e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:25.484899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.484839 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:25.484899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.484862 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:25.486134 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.486108 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs" pod "4529cf2d-afc8-4338-b21e-65bdf6cdf76e" (UID: "4529cf2d-afc8-4338-b21e-65bdf6cdf76e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:09:25.486309 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.486289 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk" pod "4529cf2d-afc8-4338-b21e-65bdf6cdf76e" (UID: "4529cf2d-afc8-4338-b21e-65bdf6cdf76e"). InnerVolumeSpecName "kube-api-access-67hfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:25.486490 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.486470 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm" pod "4529cf2d-afc8-4338-b21e-65bdf6cdf76e" (UID: "4529cf2d-afc8-4338-b21e-65bdf6cdf76e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:25.585950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.585919 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:25.585950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.585945 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:25.585950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.585954 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:25.586218 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:25.585963 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hfk\" (UniqueName: \"kubernetes.io/projected/4529cf2d-afc8-4338-b21e-65bdf6cdf76e-kube-api-access-67hfk\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:26.040289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.040245 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:26.250617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.250553 5693 generic.go:356] "Generic (PLEG): container finished" podID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerID="b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed" exitCode=0 Oct 30 14:09:26.250769 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.250680 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" Oct 30 14:09:26.256468 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.256439 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" event={"ID":"4529cf2d-afc8-4338-b21e-65bdf6cdf76e","Type":"ContainerDied","Data":"b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed"} Oct 30 14:09:26.256468 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.256476 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf" event={"ID":"4529cf2d-afc8-4338-b21e-65bdf6cdf76e","Type":"ContainerDied","Data":"1a8e26c896f95f90ec47abd2aeb71c30766e08929a80997cd1afb4a2e49c22da"} Oct 30 14:09:26.256730 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.256493 5693 scope.go:122] "RemoveContainer" containerID="b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed" Oct 30 14:09:26.266784 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.266759 5693 scope.go:122] "RemoveContainer" containerID="b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed" Oct 30 14:09:26.267076 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:26.267055 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed\": container with ID starting with b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed not found: ID does not exist" containerID="b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed" Oct 30 14:09:26.267119 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.267085 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed"} err="failed to get container status \"b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed\": rpc error: code = NotFound desc = could not find container \"b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed\": container with ID starting with b890d3a27fa4f0a5c215e05082f2a6c899e32ac747341a31c3aa63082ff20aed not found: ID does not exist" Oct 30 14:09:26.307357 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.307292 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf"] Oct 30 14:09:26.318887 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:26.318857 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-2-w4-9bf8ad-kserve-794785859d-4szsf"] Oct 30 14:09:27.260135 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:27.260090 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:09:27.383860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:27.383824 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:28.255015 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:28.254982 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" path="/var/lib/kubelet/pods/4529cf2d-afc8-4338-b21e-65bdf6cdf76e/volumes" Oct 30 14:09:28.296912 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:28.296869 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" podUID="3789b872-e323-4b81-a952-6783a7209ece" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:09:29.119521 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:29.119484 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:30.759069 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:30.759034 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:32.306119 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:32.306064 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:34.093340 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:34.093301 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:34.991317 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:34.991287 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:35.742827 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:35.742790 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:37.260133 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:37.260091 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:09:37.418780 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:37.418676 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:38.296927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:38.296887 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" podUID="3789b872-e323-4b81-a952-6783a7209ece" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:09:39.225095 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:39.225057 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:40.811412 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:40.811374 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:41.243771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.243738 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 14:09:41.290912 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.290884 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 14:09:41.360126 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.360042 5693 generic.go:356] "Generic (PLEG): container finished" podID="3789b872-e323-4b81-a952-6783a7209ece" containerID="d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e" exitCode=0 Oct 30 14:09:41.360126 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.360087 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" Oct 30 14:09:41.360323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.360119 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" event={"ID":"3789b872-e323-4b81-a952-6783a7209ece","Type":"ContainerDied","Data":"d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e"} Oct 30 14:09:41.360323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.360173 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t" event={"ID":"3789b872-e323-4b81-a952-6783a7209ece","Type":"ContainerDied","Data":"bf4b710cd3980ef55f3b23a9b8ea8de6df2aa0e57916756ba21c52010a959e1b"} Oct 30 14:09:41.360323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.360203 5693 scope.go:122] "RemoveContainer" containerID="d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e" Oct 30 14:09:41.361794 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.361767 5693 generic.go:356] "Generic (PLEG): container finished" podID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerID="0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706" exitCode=0 Oct 30 14:09:41.361932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.361806 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" Oct 30 14:09:41.361932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.361865 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" event={"ID":"be62a5bc-0e09-4d0e-a514-14b28e9b98df","Type":"ContainerDied","Data":"0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706"} Oct 30 14:09:41.361932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.361919 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5" event={"ID":"be62a5bc-0e09-4d0e-a514-14b28e9b98df","Type":"ContainerDied","Data":"b2187312e3363f2ee22790c5ed87330f51717dd17e3faf3918e1bb139dcbf2f2"} Oct 30 14:09:41.370416 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.370383 5693 scope.go:122] "RemoveContainer" containerID="d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e" Oct 30 14:09:41.370725 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:41.370698 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e\": container with ID starting with d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e not found: ID does not exist" containerID="d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e" Oct 30 14:09:41.370829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.370731 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e"} err="failed to get container status \"d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e\": rpc error: code = NotFound desc = could not find container \"d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e\": container with ID starting with d2e4996be770465455bd2034d143a034223cc593540bc48bf70c074cd37df12e not found: ID does not exist" Oct 30 14:09:41.370829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.370748 5693 scope.go:122] "RemoveContainer" containerID="0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706" Oct 30 14:09:41.382546 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.382521 5693 scope.go:122] "RemoveContainer" containerID="0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706" Oct 30 14:09:41.382875 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:41.382855 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706\": container with ID starting with 0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706 not found: ID does not exist" containerID="0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706" Oct 30 14:09:41.382962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.382884 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706"} err="failed to get container status \"0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706\": rpc error: code = NotFound desc = could not find container \"0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706\": container with ID starting with 0f686397fa2039588756d72e0148009a5fb837b15acc38559d1e793be2ac3706 not found: ID does not exist" Oct 30 14:09:41.399716 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.399692 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume\") pod \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " Oct 30 14:09:41.399842 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.399806 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5\" (UniqueName: \"kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5\") pod \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\" (UID: \"be62a5bc-0e09-4d0e-a514-14b28e9b98df\") " Oct 30 14:09:41.399895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.399879 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9\" (UniqueName: \"kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9\") pod \"3789b872-e323-4b81-a952-6783a7209ece\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " Oct 30 14:09:41.399962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.399944 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume\") pod \"3789b872-e323-4b81-a952-6783a7209ece\" (UID: \"3789b872-e323-4b81-a952-6783a7209ece\") " Oct 30 14:09:41.400089 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.400066 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume" pod "be62a5bc-0e09-4d0e-a514-14b28e9b98df" (UID: "be62a5bc-0e09-4d0e-a514-14b28e9b98df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:41.400292 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.400271 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume" pod "3789b872-e323-4b81-a952-6783a7209ece" (UID: "3789b872-e323-4b81-a952-6783a7209ece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:41.400478 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.400460 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be62a5bc-0e09-4d0e-a514-14b28e9b98df-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:41.400540 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.400484 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3789b872-e323-4b81-a952-6783a7209ece-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:41.402033 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.402012 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9" pod "3789b872-e323-4b81-a952-6783a7209ece" (UID: "3789b872-e323-4b81-a952-6783a7209ece"). InnerVolumeSpecName "kube-api-access-ssjn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:41.402135 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.402114 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5" pod "be62a5bc-0e09-4d0e-a514-14b28e9b98df" (UID: "be62a5bc-0e09-4d0e-a514-14b28e9b98df"). InnerVolumeSpecName "kube-api-access-ztnt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:41.501665 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.501630 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztnt5\" (UniqueName: \"kubernetes.io/projected/be62a5bc-0e09-4d0e-a514-14b28e9b98df-kube-api-access-ztnt5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:41.501665 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.501660 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjn9\" (UniqueName: \"kubernetes.io/projected/3789b872-e323-4b81-a952-6783a7209ece-kube-api-access-ssjn9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:09:41.705088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.705054 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t"] Oct 30 14:09:41.716419 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.716384 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9bf8ad-6989b9554-p465t"] Oct 30 14:09:41.751863 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.751825 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5"] Oct 30 14:09:41.754909 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:41.754883 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9bf8ad-69b889676c-jm4z5"] Oct 30 14:09:42.255036 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:42.255005 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3789b872-e323-4b81-a952-6783a7209ece" path="/var/lib/kubelet/pods/3789b872-e323-4b81-a952-6783a7209ece/volumes" Oct 30 14:09:42.255422 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:42.255399 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" path="/var/lib/kubelet/pods/be62a5bc-0e09-4d0e-a514-14b28e9b98df/volumes" Oct 30 14:09:42.420221 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:42.420176 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:44.154321 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:44.154240 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:45.798059 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:45.798025 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:47.391205 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:47.391165 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:47.634063 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:47.634026 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:47.907796 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:47.907758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:48.065777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:48.065743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:49.230998 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:49.230966 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:49.960995 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:49.960955 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:50.237809 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:50.237774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:50.825950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:50.825914 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:52.434812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:52.434772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:53.056017 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:09:53.055983 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:09:53.897779 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:53.897741 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:54.438540 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:54.438512 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:54.588376 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:54.588347 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:54.733295 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:54.733252 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:55.037096 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:55.036961 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:56.283718 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:56.283684 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:56.602716 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:56.602628 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:56.749310 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:56.749277 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:57.840761 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:57.840722 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:59.267993 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:59.267962 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:59.550437 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:09:59.550356 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:01.124403 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:01.124364 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:01.411988 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:01.411887 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:01.580254 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:01.580216 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:01.784564 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:01.784533 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:02.981630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:02.981557 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:04.728916 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:04.728871 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:06.502197 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:06.502149 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:07.184013 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:07.183975 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g"] Oct 30 14:10:07.269347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:07.269309 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:08.097750 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:08.097714 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:09.695166 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:09.695091 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:11.469330 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:11.469275 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:13.120714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:13.120680 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:14.795196 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:14.795158 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:16.532812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:16.532755 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:18.331035 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:18.330997 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:18.750533 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:18.750458 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:20.126056 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:20.126010 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:21.681446 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:21.681401 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:22.208348 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.208299 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" podUID="7014e751-7cca-4fe1-bfc4-99170da7256a" containerName="main" containerID="cri-o://4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e" gracePeriod=45 Oct 30 14:10:22.481175 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.481150 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:10:22.561663 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561624 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache\") pod \"7014e751-7cca-4fe1-bfc4-99170da7256a\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " Oct 30 14:10:22.561832 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561722 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir\") pod \"7014e751-7cca-4fe1-bfc4-99170da7256a\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " Oct 30 14:10:22.561832 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561762 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs\") pod \"7014e751-7cca-4fe1-bfc4-99170da7256a\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " Oct 30 14:10:22.561958 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561847 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj\" (UniqueName: \"kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj\") pod \"7014e751-7cca-4fe1-bfc4-99170da7256a\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " Oct 30 14:10:22.561958 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561879 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache" pod "7014e751-7cca-4fe1-bfc4-99170da7256a" (UID: "7014e751-7cca-4fe1-bfc4-99170da7256a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:10:22.561958 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561951 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir" pod "7014e751-7cca-4fe1-bfc4-99170da7256a" (UID: "7014e751-7cca-4fe1-bfc4-99170da7256a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:10:22.562137 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.561959 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home\") pod \"7014e751-7cca-4fe1-bfc4-99170da7256a\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " Oct 30 14:10:22.562137 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.562012 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm\") pod \"7014e751-7cca-4fe1-bfc4-99170da7256a\" (UID: \"7014e751-7cca-4fe1-bfc4-99170da7256a\") " Oct 30 14:10:22.562252 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.562134 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home" pod "7014e751-7cca-4fe1-bfc4-99170da7256a" (UID: "7014e751-7cca-4fe1-bfc4-99170da7256a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:10:22.562706 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.562682 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:22.562706 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.562705 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:22.562874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.562717 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:22.564105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.564075 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm" pod "7014e751-7cca-4fe1-bfc4-99170da7256a" (UID: "7014e751-7cca-4fe1-bfc4-99170da7256a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:10:22.564213 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.564135 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs" pod "7014e751-7cca-4fe1-bfc4-99170da7256a" (UID: "7014e751-7cca-4fe1-bfc4-99170da7256a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:10:22.564213 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.564152 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj" pod "7014e751-7cca-4fe1-bfc4-99170da7256a" (UID: "7014e751-7cca-4fe1-bfc4-99170da7256a"). InnerVolumeSpecName "kube-api-access-25dhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:10:22.656755 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.656716 5693 generic.go:356] "Generic (PLEG): container finished" podID="7014e751-7cca-4fe1-bfc4-99170da7256a" containerID="4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e" exitCode=0 Oct 30 14:10:22.656906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.656759 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" Oct 30 14:10:22.656906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.656800 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" event={"ID":"7014e751-7cca-4fe1-bfc4-99170da7256a","Type":"ContainerDied","Data":"4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e"} Oct 30 14:10:22.656906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.656842 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g" event={"ID":"7014e751-7cca-4fe1-bfc4-99170da7256a","Type":"ContainerDied","Data":"2abcfb7a0343de5ccd8e4fc218d7f80c6061ec4b1c22baffe46fcf42d60a7c48"} Oct 30 14:10:22.656906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.656873 5693 scope.go:122] "RemoveContainer" containerID="4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e" Oct 30 14:10:22.663968 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.663941 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25dhj\" (UniqueName: \"kubernetes.io/projected/7014e751-7cca-4fe1-bfc4-99170da7256a-kube-api-access-25dhj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:22.664090 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.663974 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7014e751-7cca-4fe1-bfc4-99170da7256a-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:22.664090 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.663989 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7014e751-7cca-4fe1-bfc4-99170da7256a-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:22.667502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.667485 5693 scope.go:122] "RemoveContainer" containerID="4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e" Oct 30 14:10:22.667821 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:10:22.667800 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e\": container with ID starting with 4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e not found: ID does not exist" containerID="4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e" Oct 30 14:10:22.667902 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.667826 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e"} err="failed to get container status \"4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e\": rpc error: code = NotFound desc = could not find container \"4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e\": container with ID starting with 4a4ec6535f196692f390b7ff11173e48d6e39f4f33221819ca40464d5c011c3e not found: ID does not exist" Oct 30 14:10:22.708685 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.708655 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g"] Oct 30 14:10:22.712447 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:22.712390 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3f24fd/rate-test-model-3f24fd-kserve-6bd68b8c49-b827g"] Oct 30 14:10:23.056674 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:10:23.056636 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:10:23.328238 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:23.328151 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:24.255296 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:24.255263 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7014e751-7cca-4fe1-bfc4-99170da7256a" path="/var/lib/kubelet/pods/7014e751-7cca-4fe1-bfc4-99170da7256a/volumes" Oct 30 14:10:25.019558 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:25.019526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:26.805387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:26.805351 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:28.601821 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:28.601743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:29.308306 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:29.308271 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm"] Oct 30 14:10:29.479003 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:29.478967 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:29.633191 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:29.633107 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:30.373731 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:30.373697 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:32.017576 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:32.017539 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:33.713697 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:33.713662 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:34.308980 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:34.308934 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerName="payload-processing" containerID="cri-o://1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e" gracePeriod=25 Oct 30 14:10:35.416545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:35.416459 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:37.176529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:37.176492 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:38.842880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:38.842841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:40.085759 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.085723 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg"] Oct 30 14:10:40.086266 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.085975 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" podUID="98d025ec-a9d7-4b3b-884d-80e52a801a92" containerName="maas-api" containerID="cri-o://66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395" gracePeriod=30 Oct 30 14:10:40.200653 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.200624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:40.382092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.382066 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 14:10:40.506587 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.506555 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls\") pod \"98d025ec-a9d7-4b3b-884d-80e52a801a92\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " Oct 30 14:10:40.506813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.506711 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca\") pod \"98d025ec-a9d7-4b3b-884d-80e52a801a92\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " Oct 30 14:10:40.506894 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.506816 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj\" (UniqueName: \"kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj\") pod \"98d025ec-a9d7-4b3b-884d-80e52a801a92\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " Oct 30 14:10:40.506894 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.506842 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs\") pod \"98d025ec-a9d7-4b3b-884d-80e52a801a92\" (UID: \"98d025ec-a9d7-4b3b-884d-80e52a801a92\") " Oct 30 14:10:40.507184 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.507152 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca" pod "98d025ec-a9d7-4b3b-884d-80e52a801a92" (UID: "98d025ec-a9d7-4b3b-884d-80e52a801a92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:10:40.509114 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.509085 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs" pod "98d025ec-a9d7-4b3b-884d-80e52a801a92" (UID: "98d025ec-a9d7-4b3b-884d-80e52a801a92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:10:40.509312 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.509293 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls" pod "98d025ec-a9d7-4b3b-884d-80e52a801a92" (UID: "98d025ec-a9d7-4b3b-884d-80e52a801a92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:10:40.509631 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.509574 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj" pod "98d025ec-a9d7-4b3b-884d-80e52a801a92" (UID: "98d025ec-a9d7-4b3b-884d-80e52a801a92"). InnerVolumeSpecName "kube-api-access-vb9zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:10:40.610686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.608472 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb9zj\" (UniqueName: \"kubernetes.io/projected/98d025ec-a9d7-4b3b-884d-80e52a801a92-kube-api-access-vb9zj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:40.610686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.608509 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:40.610686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.608523 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d025ec-a9d7-4b3b-884d-80e52a801a92-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:40.610686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.608536 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d025ec-a9d7-4b3b-884d-80e52a801a92-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:40.706397 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.706364 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:40.792909 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.792873 5693 generic.go:356] "Generic (PLEG): container finished" podID="98d025ec-a9d7-4b3b-884d-80e52a801a92" containerID="66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395" exitCode=0 Oct 30 14:10:40.793099 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.792917 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" Oct 30 14:10:40.793099 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.792958 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" event={"ID":"98d025ec-a9d7-4b3b-884d-80e52a801a92","Type":"ContainerDied","Data":"66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395"} Oct 30 14:10:40.793099 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.793011 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg" event={"ID":"98d025ec-a9d7-4b3b-884d-80e52a801a92","Type":"ContainerDied","Data":"6c857c9fb42d17f89a0667a5ed7e0b2cc073d3ed7ab13c100bb12c819057cdba"} Oct 30 14:10:40.793099 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.793037 5693 scope.go:122] "RemoveContainer" containerID="66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395" Oct 30 14:10:40.804816 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.804797 5693 scope.go:122] "RemoveContainer" containerID="66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395" Oct 30 14:10:40.805143 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:10:40.805122 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395\": container with ID starting with 66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395 not found: ID does not exist" containerID="66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395" Oct 30 14:10:40.805213 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.805151 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395"} err="failed to get container status \"66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395\": rpc error: code = NotFound desc = could not find container \"66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395\": container with ID starting with 66be5c026f195009fd5f1330795feb7c3be50dc8bfa03f7c5a29d05285ea8395 not found: ID does not exist" Oct 30 14:10:40.849561 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.849500 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg"] Oct 30 14:10:40.859584 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.859151 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-74589c-75f5df8d8-xsfzg"] Oct 30 14:10:40.861733 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:40.860285 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:41.300120 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:41.300085 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:42.015303 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:42.015271 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:42.166349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:42.166308 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:42.254679 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:42.254647 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d025ec-a9d7-4b3b-884d-80e52a801a92" path="/var/lib/kubelet/pods/98d025ec-a9d7-4b3b-884d-80e52a801a92/volumes" Oct 30 14:10:42.629804 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:42.629758 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:44.153033 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:44.152911 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:10:44.302478 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:44.302425 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:45.925723 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:45.925685 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:47.631280 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:47.631247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:49.286045 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:49.286010 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:49.442654 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:49.442619 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:49.758978 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:49.758941 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:51.239343 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:51.239289 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:52.921801 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:52.921746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:53.057084 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:10:53.057035 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:10:54.152915 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.152875 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:10:54.583494 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.583467 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 14:10:54.593525 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.593486 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:54.711731 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.711594 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2\" (UniqueName: \"kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2\") pod \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " Oct 30 14:10:54.711895 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.711819 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume\") pod \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\" (UID: \"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5\") " Oct 30 14:10:54.712218 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.712190 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume" pod "39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" (UID: "39f8754c-1a5a-4ea0-86eb-ad4c31e876f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:10:54.713836 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.713815 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2" pod "39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" (UID: "39f8754c-1a5a-4ea0-86eb-ad4c31e876f5"). InnerVolumeSpecName "kube-api-access-94qn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:10:54.813751 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.813713 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:54.813751 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.813745 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qn2\" (UniqueName: \"kubernetes.io/projected/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5-kube-api-access-94qn2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:10:54.893362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.893323 5693 generic.go:356] "Generic (PLEG): container finished" podID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerID="1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e" exitCode=0 Oct 30 14:10:54.893538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.893368 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" Oct 30 14:10:54.893538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.893401 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" event={"ID":"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5","Type":"ContainerDied","Data":"1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e"} Oct 30 14:10:54.893538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.893458 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm" event={"ID":"39f8754c-1a5a-4ea0-86eb-ad4c31e876f5","Type":"ContainerDied","Data":"c6ed2bc9f563bc3401d0b3a5c748bf6a564d58b467413babffac3596073ab82f"} Oct 30 14:10:54.893538 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.893481 5693 scope.go:122] "RemoveContainer" containerID="1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e" Oct 30 14:10:54.908946 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.908910 5693 scope.go:122] "RemoveContainer" containerID="1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e" Oct 30 14:10:54.909437 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:10:54.909414 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e\": container with ID starting with 1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e not found: ID does not exist" containerID="1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e" Oct 30 14:10:54.909528 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.909444 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e"} err="failed to get container status \"1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e\": rpc error: code = NotFound desc = could not find container \"1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e\": container with ID starting with 1d4b2d38ed104baecd61813536dd592ddfdbb15cc271dfed7d1f21d0998da48e not found: ID does not exist" Oct 30 14:10:54.936606 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.936571 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:54.965706 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.945336 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm"] Oct 30 14:10:54.968483 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:54.966851 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-74589c-68cbd94f5b-p9dkm"] Oct 30 14:10:56.225311 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:56.225280 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:56.264710 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:56.264677 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" path="/var/lib/kubelet/pods/39f8754c-1a5a-4ea0-86eb-ad4c31e876f5/volumes" Oct 30 14:10:56.531839 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:56.531750 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf"] Oct 30 14:10:56.608506 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:56.608471 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:58.180214 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:58.180180 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:59.397270 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:59.397236 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:59.894578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:10:59.894538 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:00.877001 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:00.876952 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:01.015386 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:01.015350 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:01.571198 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:01.571153 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:03.126384 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:03.126344 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:04.747542 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:04.747498 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:06.409030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:06.409000 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:07.349222 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:07.349192 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:07.484945 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:07.484916 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:07.719190 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:07.719065 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c"] Oct 30 14:11:07.720793 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:07.720746 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" podUID="7b98b358-3b68-43b6-93db-d585fb5a2d82" containerName="maas-api" containerID="cri-o://1ab136504c7126b7c0c04e446f1cd1fb61fca0d53427e7dd59bef739a85352d4" gracePeriod=30 Oct 30 14:11:07.827082 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:07.827050 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:08.034367 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.034335 5693 generic.go:356] "Generic (PLEG): container finished" podID="7b98b358-3b68-43b6-93db-d585fb5a2d82" containerID="1ab136504c7126b7c0c04e446f1cd1fb61fca0d53427e7dd59bef739a85352d4" exitCode=0 Oct 30 14:11:08.034533 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.034414 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" event={"ID":"7b98b358-3b68-43b6-93db-d585fb5a2d82","Type":"ContainerDied","Data":"1ab136504c7126b7c0c04e446f1cd1fb61fca0d53427e7dd59bef739a85352d4"} Oct 30 14:11:08.051764 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.051736 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 14:11:08.122140 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.122110 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs\") pod \"7b98b358-3b68-43b6-93db-d585fb5a2d82\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " Oct 30 14:11:08.122332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.122151 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls\") pod \"7b98b358-3b68-43b6-93db-d585fb5a2d82\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " Oct 30 14:11:08.122332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.122225 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz\" (UniqueName: \"kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz\") pod \"7b98b358-3b68-43b6-93db-d585fb5a2d82\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " Oct 30 14:11:08.122453 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.122379 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca\") pod \"7b98b358-3b68-43b6-93db-d585fb5a2d82\" (UID: \"7b98b358-3b68-43b6-93db-d585fb5a2d82\") " Oct 30 14:11:08.123360 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.123333 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca" pod "7b98b358-3b68-43b6-93db-d585fb5a2d82" (UID: "7b98b358-3b68-43b6-93db-d585fb5a2d82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:08.124950 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.124918 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs" pod "7b98b358-3b68-43b6-93db-d585fb5a2d82" (UID: "7b98b358-3b68-43b6-93db-d585fb5a2d82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:08.125055 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.124944 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls" pod "7b98b358-3b68-43b6-93db-d585fb5a2d82" (UID: "7b98b358-3b68-43b6-93db-d585fb5a2d82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:08.125103 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.125060 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz" pod "7b98b358-3b68-43b6-93db-d585fb5a2d82" (UID: "7b98b358-3b68-43b6-93db-d585fb5a2d82"). InnerVolumeSpecName "kube-api-access-n7pgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:08.225470 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.225429 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:08.225470 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.225468 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b98b358-3b68-43b6-93db-d585fb5a2d82-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:08.225734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.225487 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7pgz\" (UniqueName: \"kubernetes.io/projected/7b98b358-3b68-43b6-93db-d585fb5a2d82-kube-api-access-n7pgz\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:08.225734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.225504 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b98b358-3b68-43b6-93db-d585fb5a2d82-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:08.392911 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.392875 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:08.534138 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.534103 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:08.680825 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:08.680743 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:09.027351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:09.027320 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:09.042456 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:09.042428 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" Oct 30 14:11:09.042593 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:09.042465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c" event={"ID":"7b98b358-3b68-43b6-93db-d585fb5a2d82","Type":"ContainerDied","Data":"dbb217b30409e5aafecd3c36bc6d270f0714a7563392e7e4e9d56d4ca25315a5"} Oct 30 14:11:09.042593 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:09.042524 5693 scope.go:122] "RemoveContainer" containerID="1ab136504c7126b7c0c04e446f1cd1fb61fca0d53427e7dd59bef739a85352d4" Oct 30 14:11:09.092320 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:09.092286 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c"] Oct 30 14:11:09.104470 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:09.104440 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-98f1cc-77b8497c9f-c578c"] Oct 30 14:11:10.117411 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:10.117353 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:10.290622 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:10.290522 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b98b358-3b68-43b6-93db-d585fb5a2d82" path="/var/lib/kubelet/pods/7b98b358-3b68-43b6-93db-d585fb5a2d82/volumes" Oct 30 14:11:10.628096 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:10.628056 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:10.829796 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:10.829751 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:10.973369 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:10.973267 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:11.248925 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.248893 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:11.554577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.554490 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" podUID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" containerName="main" containerID="cri-o://1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f" gracePeriod=45 Oct 30 14:11:11.598208 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.598172 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" podUID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" containerName="main" probeResult="failure" output="Get \"https://10.134.1.30:8000/ready\": dial tcp 10.134.1.30:8000: connect: connection refused" Oct 30 14:11:11.823532 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.823468 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:11:11.977971 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.977937 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm\") pod \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " Oct 30 14:11:11.978160 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs\") pod \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " Oct 30 14:11:11.978160 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978068 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache\") pod \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " Oct 30 14:11:11.978160 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978126 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj\" (UniqueName: \"kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj\") pod \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " Oct 30 14:11:11.978320 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978208 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home\") pod \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " Oct 30 14:11:11.978320 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978235 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir\") pod \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\" (UID: \"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14\") " Oct 30 14:11:11.978442 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978391 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache" pod "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" (UID: "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:11.978511 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978482 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home" pod "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" (UID: "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:11.978573 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978553 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir" pod "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" (UID: "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:11.978850 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978834 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:11.978915 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978852 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:11.978915 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.978861 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:11.980761 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.980731 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs" pod "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" (UID: "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:11.980761 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.980734 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm" pod "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" (UID: "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:11.980899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:11.980820 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj" pod "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" (UID: "6d967b1c-4a62-49ff-bbc0-0e5900fd7d14"). InnerVolumeSpecName "kube-api-access-r6cqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:12.064989 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.064956 5693 generic.go:356] "Generic (PLEG): container finished" podID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" containerID="1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f" exitCode=0 Oct 30 14:11:12.065172 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.065008 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" Oct 30 14:11:12.065172 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.065042 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" event={"ID":"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14","Type":"ContainerDied","Data":"1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f"} Oct 30 14:11:12.065172 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.065084 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf" event={"ID":"6d967b1c-4a62-49ff-bbc0-0e5900fd7d14","Type":"ContainerDied","Data":"fcc4431921a4458e055aa509a6d41ba9050beca0b84745f86aa3e833a97d37d7"} Oct 30 14:11:12.065172 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.065102 5693 scope.go:122] "RemoveContainer" containerID="1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f" Oct 30 14:11:12.075764 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.075743 5693 scope.go:122] "RemoveContainer" containerID="1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f" Oct 30 14:11:12.076030 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:12.076009 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f\": container with ID starting with 1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f not found: ID does not exist" containerID="1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f" Oct 30 14:11:12.076101 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.076039 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f"} err="failed to get container status \"1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f\": rpc error: code = NotFound desc = could not find container \"1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f\": container with ID starting with 1abfb692f3f2ba00f3c69b1883aaf1c44a4d7d9d83aadf77586009e1dd98101f not found: ID does not exist" Oct 30 14:11:12.079718 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.079695 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:12.079718 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.079716 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:12.079859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.079725 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cqj\" (UniqueName: \"kubernetes.io/projected/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14-kube-api-access-r6cqj\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:12.119933 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.119897 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf"] Oct 30 14:11:12.126033 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.126002 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/rate-test-model-29a3f3-kserve-768949df97-dq2kf"] Oct 30 14:11:12.256641 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.256586 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" path="/var/lib/kubelet/pods/6d967b1c-4a62-49ff-bbc0-0e5900fd7d14/volumes" Oct 30 14:11:12.601679 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:12.601642 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:14.186630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:14.186547 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:15.656999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:15.656966 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:15.809408 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:15.809347 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:15.955691 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:15.955577 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:16.097189 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:16.097159 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:17.690832 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:17.690794 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:18.102875 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:18.102838 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:18.271578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:18.271542 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:18.548617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:18.547850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:19.520614 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:19.520556 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:19.690801 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:19.690761 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:21.303126 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:21.303063 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:22.992732 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:22.992697 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:23.055316 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:23.055276 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:11:24.626686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:24.626643 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:26.214362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:26.214312 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:26.851148 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:26.851107 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll"] Oct 30 14:11:26.880406 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:26.880374 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg"] Oct 30 14:11:27.009979 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:27.009952 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:27.143338 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:27.143247 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:27.915301 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:27.915264 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:29.444907 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:29.444871 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5"] Oct 30 14:11:29.555885 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:29.555852 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:29.695812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:29.695725 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:31.193478 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:31.193438 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:31.752903 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:31.752859 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx"] Oct 30 14:11:31.829287 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:31.829258 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:31.851900 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:31.851855 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerName="payload-processing" containerID="cri-o://6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7" gracePeriod=25 Oct 30 14:11:31.883770 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:31.883721 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerName="payload-pre-processing" containerID="cri-o://92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f" gracePeriod=25 Oct 30 14:11:32.905258 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:32.905218 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:34.059539 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:34.059500 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:34.059962 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:34.059500 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:34.445343 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:34.445257 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerName="payload-processing" containerID="cri-o://b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971" gracePeriod=25 Oct 30 14:11:34.628569 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:34.628526 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:36.303552 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:36.303519 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:36.868871 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:36.868831 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:37.022393 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:37.022354 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:37.507640 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:37.507607 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:38.403704 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.403653 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276"] Oct 30 14:11:38.403940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.403906 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" podUID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" containerName="maas-api" containerID="cri-o://19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275" gracePeriod=30 Oct 30 14:11:38.466211 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.466174 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" podUID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.206:8443/health\": dial tcp 10.134.0.206:8443: connect: connection refused" Oct 30 14:11:38.513819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.513785 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:38.679904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.679831 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 14:11:38.782903 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.782870 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls\") pod \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " Oct 30 14:11:38.783081 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.782929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5\" (UniqueName: \"kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5\") pod \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " Oct 30 14:11:38.783081 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.782955 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca\") pod \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " Oct 30 14:11:38.783081 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.783036 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs\") pod \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\" (UID: \"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2\") " Oct 30 14:11:38.783455 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.783421 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca" pod "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" (UID: "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:38.783554 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.783535 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:38.785258 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.785227 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5" pod "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" (UID: "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2"). InnerVolumeSpecName "kube-api-access-w4wv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:38.785258 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.785247 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs" pod "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" (UID: "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:38.785424 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.785276 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls" pod "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" (UID: "d975ade9-c5cf-4335-9d2f-3ead72ae9bb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:38.884749 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.884704 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:38.884749 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.884750 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:38.884749 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:38.884765 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4wv5\" (UniqueName: \"kubernetes.io/projected/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2-kube-api-access-w4wv5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:39.002561 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.002519 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:39.204725 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.204692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:39.278463 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.278376 5693 generic.go:356] "Generic (PLEG): container finished" podID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" containerID="19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275" exitCode=0 Oct 30 14:11:39.278463 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.278430 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" event={"ID":"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2","Type":"ContainerDied","Data":"19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275"} Oct 30 14:11:39.278463 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.278436 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" Oct 30 14:11:39.278776 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.278470 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276" event={"ID":"d975ade9-c5cf-4335-9d2f-3ead72ae9bb2","Type":"ContainerDied","Data":"3b0d78002d90d3ae76901ee9f124d69e78682f0a33b923ed4accbd7e70fdaf79"} Oct 30 14:11:39.278776 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.278490 5693 scope.go:122] "RemoveContainer" containerID="19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275" Oct 30 14:11:39.290228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.290208 5693 scope.go:122] "RemoveContainer" containerID="19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275" Oct 30 14:11:39.290492 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:39.290473 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275\": container with ID starting with 19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275 not found: ID does not exist" containerID="19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275" Oct 30 14:11:39.290547 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.290502 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275"} err="failed to get container status \"19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275\": rpc error: code = NotFound desc = could not find container \"19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275\": container with ID starting with 19085003e99e08398ca4ba5b6aac235dcf606cb8029fe3d90306bbd624123275 not found: ID does not exist" Oct 30 14:11:39.349228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.349192 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276"] Oct 30 14:11:39.357817 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.357787 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3f24fd-556c4ff7bc-xs276"] Oct 30 14:11:39.451162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.451129 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:39.821006 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:39.820975 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:40.255207 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:40.255175 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" path="/var/lib/kubelet/pods/d975ade9-c5cf-4335-9d2f-3ead72ae9bb2/volumes" Oct 30 14:11:41.115300 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:41.115263 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:42.393985 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:42.393952 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk"] Oct 30 14:11:42.422505 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:42.422470 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm"] Oct 30 14:11:42.561212 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:42.561176 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:42.702656 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:42.702570 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:44.060383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:44.060341 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:44.060920 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:44.060415 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:44.411301 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:44.411180 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:45.501812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:45.501760 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:45.638114 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:45.638081 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:45.837983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:45.837904 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:46.024625 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:46.024558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:46.774732 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:46.774680 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" podUID="fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" containerName="main" containerID="cri-o://b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248" gracePeriod=45 Oct 30 14:11:47.053053 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.053027 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:11:47.203351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203311 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx\" (UniqueName: \"kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx\") pod \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " Oct 30 14:11:47.203556 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203415 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home\") pod \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " Oct 30 14:11:47.203556 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203535 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir\") pod \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " Oct 30 14:11:47.203708 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203591 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm\") pod \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " Oct 30 14:11:47.203708 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203675 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache\") pod \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " Oct 30 14:11:47.203824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203717 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs\") pod \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\" (UID: \"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb\") " Oct 30 14:11:47.203824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203716 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home" pod "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" (UID: "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:47.203928 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203830 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir" pod "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" (UID: "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:47.203983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.203944 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache" pod "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" (UID: "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:47.204360 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.204336 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.204484 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.204385 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.204484 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.204399 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.205938 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.205913 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm" pod "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" (UID: "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:47.205938 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.205924 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs" pod "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" (UID: "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:47.206087 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.206065 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx" pod "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" (UID: "fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb"). InnerVolumeSpecName "kube-api-access-6ccxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:47.306647 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.306480 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccxx\" (UniqueName: \"kubernetes.io/projected/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-kube-api-access-6ccxx\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.307200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.307177 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.307323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.307215 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.310523 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.308291 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:47.332028 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.331991 5693 generic.go:356] "Generic (PLEG): container finished" podID="fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" containerID="b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248" exitCode=0 Oct 30 14:11:47.332028 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.332029 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" Oct 30 14:11:47.332264 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.332073 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" event={"ID":"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb","Type":"ContainerDied","Data":"b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248"} Oct 30 14:11:47.332264 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.332123 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx" event={"ID":"fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb","Type":"ContainerDied","Data":"0359b6fd10cb68f8d251afa40285429c0b6a1508735e0e32c61f6140b1c70377"} Oct 30 14:11:47.332264 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.332143 5693 scope.go:122] "RemoveContainer" containerID="b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248" Oct 30 14:11:47.344148 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.344119 5693 scope.go:122] "RemoveContainer" containerID="b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248" Oct 30 14:11:47.344447 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:47.344427 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248\": container with ID starting with b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248 not found: ID does not exist" containerID="b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248" Oct 30 14:11:47.344512 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.344453 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248"} err="failed to get container status \"b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248\": rpc error: code = NotFound desc = could not find container \"b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248\": container with ID starting with b36a632b47b6e296cdc597ae92bf738fe904eb493df9f3d50043875135f27248 not found: ID does not exist" Oct 30 14:11:47.388579 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.388552 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx"] Oct 30 14:11:47.394654 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.394629 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" containerName="payload-processing" containerID="cri-o://91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0" gracePeriod=25 Oct 30 14:11:47.400328 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.400304 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/sub-test-model-22af91-kserve-68648b8c7f-f2jbx"] Oct 30 14:11:47.423098 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.423045 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerName="payload-pre-processing" containerID="cri-o://dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6" gracePeriod=25 Oct 30 14:11:47.425146 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.425120 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6"] Oct 30 14:11:47.425337 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.425316 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" podUID="70c6f506-deae-4d9e-85c7-f4507adba4b2" containerName="maas-api" containerID="cri-o://660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331" gracePeriod=30 Oct 30 14:11:47.465811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.465772 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:47.570277 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.570169 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58"] Oct 30 14:11:47.570463 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.570425 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" containerID="cri-o://c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b" gracePeriod=30 Oct 30 14:11:47.623577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.623547 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:47.684931 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.684903 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 14:11:47.766323 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.766276 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:47.819289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.813230 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs\") pod \"70c6f506-deae-4d9e-85c7-f4507adba4b2\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " Oct 30 14:11:47.819289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.813573 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca\") pod \"70c6f506-deae-4d9e-85c7-f4507adba4b2\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " Oct 30 14:11:47.819289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.813641 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls\") pod \"70c6f506-deae-4d9e-85c7-f4507adba4b2\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " Oct 30 14:11:47.819289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.813704 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks\" (UniqueName: \"kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks\") pod \"70c6f506-deae-4d9e-85c7-f4507adba4b2\" (UID: \"70c6f506-deae-4d9e-85c7-f4507adba4b2\") " Oct 30 14:11:47.819289 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.814375 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca" pod "70c6f506-deae-4d9e-85c7-f4507adba4b2" (UID: "70c6f506-deae-4d9e-85c7-f4507adba4b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:47.826964 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.826730 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs" pod "70c6f506-deae-4d9e-85c7-f4507adba4b2" (UID: "70c6f506-deae-4d9e-85c7-f4507adba4b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:47.826964 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.826782 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks" pod "70c6f506-deae-4d9e-85c7-f4507adba4b2" (UID: "70c6f506-deae-4d9e-85c7-f4507adba4b2"). InnerVolumeSpecName "kube-api-access-slfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:47.826964 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.826814 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls" pod "70c6f506-deae-4d9e-85c7-f4507adba4b2" (UID: "70c6f506-deae-4d9e-85c7-f4507adba4b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:47.917783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.917718 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.917783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.917778 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfks\" (UniqueName: \"kubernetes.io/projected/70c6f506-deae-4d9e-85c7-f4507adba4b2-kube-api-access-slfks\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.918054 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.917796 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c6f506-deae-4d9e-85c7-f4507adba4b2-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:47.918054 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:47.917813 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f506-deae-4d9e-85c7-f4507adba4b2-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:48.210991 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.210906 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:48.256736 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.256665 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" path="/var/lib/kubelet/pods/fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb/volumes" Oct 30 14:11:48.347813 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.347780 5693 generic.go:356] "Generic (PLEG): container finished" podID="70c6f506-deae-4d9e-85c7-f4507adba4b2" containerID="660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331" exitCode=0 Oct 30 14:11:48.348020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.347842 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" Oct 30 14:11:48.348020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.347861 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" event={"ID":"70c6f506-deae-4d9e-85c7-f4507adba4b2","Type":"ContainerDied","Data":"660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331"} Oct 30 14:11:48.348020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.347926 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6" event={"ID":"70c6f506-deae-4d9e-85c7-f4507adba4b2","Type":"ContainerDied","Data":"441a19b833e121b1c2812cb2d4e22a06f4c78b0f791358e59f73cf12121bcee8"} Oct 30 14:11:48.348020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.347956 5693 scope.go:122] "RemoveContainer" containerID="660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331" Oct 30 14:11:48.370450 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.370358 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:48.373892 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.373849 5693 scope.go:122] "RemoveContainer" containerID="660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331" Oct 30 14:11:48.374297 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:48.374268 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331\": container with ID starting with 660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331 not found: ID does not exist" containerID="660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331" Oct 30 14:11:48.374410 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.374304 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331"} err="failed to get container status \"660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331\": rpc error: code = NotFound desc = could not find container \"660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331\": container with ID starting with 660006ca8598751ca8359bd34ecbff1096fb72bb533cd0ab57851e66dcc87331 not found: ID does not exist" Oct 30 14:11:48.399114 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.399077 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6"] Oct 30 14:11:48.410421 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.410393 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cad2c6-5b6896f797-x2gb6"] Oct 30 14:11:48.841949 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:48.841912 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:49.002893 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:49.002855 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:50.255201 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:50.255161 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c6f506-deae-4d9e-85c7-f4507adba4b2" path="/var/lib/kubelet/pods/70c6f506-deae-4d9e-85c7-f4507adba4b2/volumes" Oct 30 14:11:50.412972 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:50.412900 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:52.007582 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.007547 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:52.172856 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.172830 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 14:11:52.223744 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.223711 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 14:11:52.277370 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.277327 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw\" (UniqueName: \"kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw\") pod \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " Oct 30 14:11:52.277559 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.277412 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume\") pod \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\" (UID: \"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12\") " Oct 30 14:11:52.277864 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.277837 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume" pod "5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" (UID: "5a41edd2-d1ed-4bc9-95bb-8491eab9dd12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:52.278203 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.278184 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.279716 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.279689 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw" pod "5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" (UID: "5a41edd2-d1ed-4bc9-95bb-8491eab9dd12"). InnerVolumeSpecName "kube-api-access-xmsqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:52.375284 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.375249 5693 generic.go:356] "Generic (PLEG): container finished" podID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerID="92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f" exitCode=0 Oct 30 14:11:52.375488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.375305 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" Oct 30 14:11:52.375488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.375337 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" event={"ID":"895e8e9a-af7b-4d49-9132-5c494886f00f","Type":"ContainerDied","Data":"92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f"} Oct 30 14:11:52.375488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.375401 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg" event={"ID":"895e8e9a-af7b-4d49-9132-5c494886f00f","Type":"ContainerDied","Data":"3f373cd81bc89afb0dce2464638f72abe9948fd303568714dde47384f151819d"} Oct 30 14:11:52.375488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.375433 5693 scope.go:122] "RemoveContainer" containerID="92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f" Oct 30 14:11:52.376723 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.376701 5693 generic.go:356] "Generic (PLEG): container finished" podID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerID="6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7" exitCode=0 Oct 30 14:11:52.376835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.376744 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" Oct 30 14:11:52.376835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.376782 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" event={"ID":"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12","Type":"ContainerDied","Data":"6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7"} Oct 30 14:11:52.376835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.376822 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll" event={"ID":"5a41edd2-d1ed-4bc9-95bb-8491eab9dd12","Type":"ContainerDied","Data":"85aa47dcc16c48273cf7a3396e3475857358cd8417d56fa4385862e731c0851b"} Oct 30 14:11:52.379073 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.379049 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume\") pod \"895e8e9a-af7b-4d49-9132-5c494886f00f\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " Oct 30 14:11:52.379204 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.379180 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt\" (UniqueName: \"kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt\") pod \"895e8e9a-af7b-4d49-9132-5c494886f00f\" (UID: \"895e8e9a-af7b-4d49-9132-5c494886f00f\") " Oct 30 14:11:52.379451 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.379418 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume" pod "895e8e9a-af7b-4d49-9132-5c494886f00f" (UID: "895e8e9a-af7b-4d49-9132-5c494886f00f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:52.379717 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.379640 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmsqw\" (UniqueName: \"kubernetes.io/projected/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12-kube-api-access-xmsqw\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.379717 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.379668 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895e8e9a-af7b-4d49-9132-5c494886f00f-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.381842 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.381813 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt" pod "895e8e9a-af7b-4d49-9132-5c494886f00f" (UID: "895e8e9a-af7b-4d49-9132-5c494886f00f"). InnerVolumeSpecName "kube-api-access-xm5vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:52.386154 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.386131 5693 scope.go:122] "RemoveContainer" containerID="92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f" Oct 30 14:11:52.386422 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:52.386403 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f\": container with ID starting with 92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f not found: ID does not exist" containerID="92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f" Oct 30 14:11:52.386487 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.386429 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f"} err="failed to get container status \"92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f\": rpc error: code = NotFound desc = could not find container \"92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f\": container with ID starting with 92b09ca09c3918d6080033db7aa010ecdb838b4e8b55176872086042fc11c92f not found: ID does not exist" Oct 30 14:11:52.386487 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.386448 5693 scope.go:122] "RemoveContainer" containerID="6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7" Oct 30 14:11:52.395239 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.395221 5693 scope.go:122] "RemoveContainer" containerID="6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7" Oct 30 14:11:52.395514 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:52.395491 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7\": container with ID starting with 6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7 not found: ID does not exist" containerID="6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7" Oct 30 14:11:52.395565 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.395520 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7"} err="failed to get container status \"6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7\": rpc error: code = NotFound desc = could not find container \"6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7\": container with ID starting with 6c3fb61d1cc0111b451fb700fad8cb79e3c96fd2c085e6baef4f5fe4ad4e53d7 not found: ID does not exist" Oct 30 14:11:52.426819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.426788 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll"] Oct 30 14:11:52.430649 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.430619 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3f24fd-64c469676b-cgtll"] Oct 30 14:11:52.481154 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.481124 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5vt\" (UniqueName: \"kubernetes.io/projected/895e8e9a-af7b-4d49-9132-5c494886f00f-kube-api-access-xm5vt\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.722056 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.722018 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg"] Oct 30 14:11:52.733710 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.733668 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3f24fd-5f8b84b4c5-cjqjg"] Oct 30 14:11:52.826773 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.826743 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 14:11:52.987454 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987419 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.987681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987484 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.987681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987533 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.987681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987633 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.987897 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987747 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.987897 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987882 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.987999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.987918 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:52.988042 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988006 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.988095 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988064 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.988165 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988118 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket\") pod \"6770ae55-0287-4dc9-a718-01f0e2136e8c\" (UID: \"6770ae55-0287-4dc9-a718-01f0e2136e8c\") " Oct 30 14:11:52.988223 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988162 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:52.988351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988303 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:52.988432 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988415 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:52.988496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988458 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:52.988806 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988782 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.988919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988809 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.988919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988823 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6770ae55-0287-4dc9-a718-01f0e2136e8c-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.988919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988837 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.988919 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.988851 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.989928 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.989898 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "kube-api-access-vxc6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:52.989928 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.989923 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:11:52.990095 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.989932 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:52.990157 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:52.990096 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo" pod "6770ae55-0287-4dc9-a718-01f0e2136e8c" (UID: "6770ae55-0287-4dc9-a718-01f0e2136e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:11:53.055106 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:53.055069 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:11:53.089990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.089958 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:53.089990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.089988 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:53.089990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.089997 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6770ae55-0287-4dc9-a718-01f0e2136e8c-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:53.090224 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.090005 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxc6r\" (UniqueName: \"kubernetes.io/projected/6770ae55-0287-4dc9-a718-01f0e2136e8c-kube-api-access-vxc6r\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:53.386332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.386233 5693 generic.go:356] "Generic (PLEG): container finished" podID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerID="c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b" exitCode=0 Oct 30 14:11:53.386332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.386290 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" Oct 30 14:11:53.386585 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.386319 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" event={"ID":"6770ae55-0287-4dc9-a718-01f0e2136e8c","Type":"ContainerDied","Data":"c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b"} Oct 30 14:11:53.386585 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.386396 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58" event={"ID":"6770ae55-0287-4dc9-a718-01f0e2136e8c","Type":"ContainerDied","Data":"e5103028c1a1d60364c8c789908c2fb446c40574bc92d3a742f5cb8ffb124b51"} Oct 30 14:11:53.386585 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.386426 5693 scope.go:122] "RemoveContainer" containerID="c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b" Oct 30 14:11:53.397838 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.397818 5693 scope.go:122] "RemoveContainer" containerID="2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244" Oct 30 14:11:53.408267 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.408243 5693 scope.go:122] "RemoveContainer" containerID="c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b" Oct 30 14:11:53.408547 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:53.408527 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b\": container with ID starting with c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b not found: ID does not exist" containerID="c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b" Oct 30 14:11:53.408611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.408558 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b"} err="failed to get container status \"c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b\": rpc error: code = NotFound desc = could not find container \"c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b\": container with ID starting with c2b6550f4baa3cbbf05a04f3a84e2e879fafd35ef07a200f3813be3c46ddb13b not found: ID does not exist" Oct 30 14:11:53.408611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.408577 5693 scope.go:122] "RemoveContainer" containerID="2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244" Oct 30 14:11:53.408876 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:53.408858 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244\": container with ID starting with 2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244 not found: ID does not exist" containerID="2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244" Oct 30 14:11:53.408926 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.408883 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244"} err="failed to get container status \"2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244\": rpc error: code = NotFound desc = could not find container \"2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244\": container with ID starting with 2636996e54f24a007a3ce284d51374097cf57f25cd17d8facdd5bca33e103244 not found: ID does not exist" Oct 30 14:11:53.440671 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.440637 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58"] Oct 30 14:11:53.452578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.452550 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3f24fd-openshift-default-8b7f5848f-bwq58"] Oct 30 14:11:53.705845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:53.705744 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:54.255134 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.255097 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" path="/var/lib/kubelet/pods/5a41edd2-d1ed-4bc9-95bb-8491eab9dd12/volumes" Oct 30 14:11:54.255960 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.255935 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" path="/var/lib/kubelet/pods/6770ae55-0287-4dc9-a718-01f0e2136e8c/volumes" Oct 30 14:11:54.256556 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.256536 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" path="/var/lib/kubelet/pods/895e8e9a-af7b-4d49-9132-5c494886f00f/volumes" Oct 30 14:11:54.711510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.711484 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 14:11:54.807121 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.807090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf\" (UniqueName: \"kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf\") pod \"1d0fbda9-c630-43fa-ab55-6d6275a24187\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " Oct 30 14:11:54.807318 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.807303 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume\") pod \"1d0fbda9-c630-43fa-ab55-6d6275a24187\" (UID: \"1d0fbda9-c630-43fa-ab55-6d6275a24187\") " Oct 30 14:11:54.807751 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.807723 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume" pod "1d0fbda9-c630-43fa-ab55-6d6275a24187" (UID: "1d0fbda9-c630-43fa-ab55-6d6275a24187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:54.809228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.809203 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf" pod "1d0fbda9-c630-43fa-ab55-6d6275a24187" (UID: "1d0fbda9-c630-43fa-ab55-6d6275a24187"). InnerVolumeSpecName "kube-api-access-w2blf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:54.908857 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.908828 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0fbda9-c630-43fa-ab55-6d6275a24187-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:54.908857 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:54.908854 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2blf\" (UniqueName: \"kubernetes.io/projected/1d0fbda9-c630-43fa-ab55-6d6275a24187-kube-api-access-w2blf\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:11:55.370445 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.370413 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:55.407798 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.407764 5693 generic.go:356] "Generic (PLEG): container finished" podID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerID="b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971" exitCode=0 Oct 30 14:11:55.407990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.407818 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" Oct 30 14:11:55.407990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.407851 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" event={"ID":"1d0fbda9-c630-43fa-ab55-6d6275a24187","Type":"ContainerDied","Data":"b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971"} Oct 30 14:11:55.407990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.407909 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5" event={"ID":"1d0fbda9-c630-43fa-ab55-6d6275a24187","Type":"ContainerDied","Data":"98194aee8678f9870f41d9e85cc70c7135f3d927e0bd24095b521114ff0a2b35"} Oct 30 14:11:55.407990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.407934 5693 scope.go:122] "RemoveContainer" containerID="b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971" Oct 30 14:11:55.418377 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.418359 5693 scope.go:122] "RemoveContainer" containerID="b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971" Oct 30 14:11:55.418674 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:11:55.418645 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971\": container with ID starting with b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971 not found: ID does not exist" containerID="b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971" Oct 30 14:11:55.418735 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.418681 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971"} err="failed to get container status \"b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971\": rpc error: code = NotFound desc = could not find container \"b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971\": container with ID starting with b02a69947ae0137df74b147822e002c85eb4b17249c5c5c0db9cfc426e9f2971 not found: ID does not exist" Oct 30 14:11:55.457373 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.457342 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5"] Oct 30 14:11:55.468909 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.468880 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a75d02-64966f5796-82tj5"] Oct 30 14:11:55.730940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.730903 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:55.895859 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.895821 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:55.992149 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.992060 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp"] Oct 30 14:11:55.992637 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:55.992341 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" containerID="cri-o://7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4" gracePeriod=30 Oct 30 14:11:56.088363 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:56.088331 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:56.263219 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:56.263138 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" path="/var/lib/kubelet/pods/1d0fbda9-c630-43fa-ab55-6d6275a24187/volumes" Oct 30 14:11:57.104090 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:57.104052 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9"] Oct 30 14:11:57.318835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:57.318680 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:57.321780 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:57.321747 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:11:57.363819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:57.363731 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:57.510687 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:57.510655 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:58.987060 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:58.987029 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:59.130465 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:59.130434 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:59.515681 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:11:59.515645 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:00.245352 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.245317 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7"] Oct 30 14:12:00.245845 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.245523 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" podUID="7878c52d-f84f-4d12-8bdd-937eb5a9644a" containerName="maas-api" containerID="cri-o://486d52d6f06843d4364aa99adb8611b84a30d2a85aaa8cded0767989f1c65d33" gracePeriod=30 Oct 30 14:12:00.351876 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.351847 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:00.442669 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.442590 5693 generic.go:356] "Generic (PLEG): container finished" podID="7878c52d-f84f-4d12-8bdd-937eb5a9644a" containerID="486d52d6f06843d4364aa99adb8611b84a30d2a85aaa8cded0767989f1c65d33" exitCode=0 Oct 30 14:12:00.442841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.442641 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" event={"ID":"7878c52d-f84f-4d12-8bdd-937eb5a9644a","Type":"ContainerDied","Data":"486d52d6f06843d4364aa99adb8611b84a30d2a85aaa8cded0767989f1c65d33"} Oct 30 14:12:00.532146 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.532115 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 14:12:00.579164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.579122 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf\" (UniqueName: \"kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf\") pod \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " Oct 30 14:12:00.579376 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.579238 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca\") pod \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " Oct 30 14:12:00.579376 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.579363 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\") pod \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " Oct 30 14:12:00.579525 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.579396 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs\") pod \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\" (UID: \"7878c52d-f84f-4d12-8bdd-937eb5a9644a\") " Oct 30 14:12:00.580020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.579984 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca" pod "7878c52d-f84f-4d12-8bdd-937eb5a9644a" (UID: "7878c52d-f84f-4d12-8bdd-937eb5a9644a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:00.581957 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.581928 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls" pod "7878c52d-f84f-4d12-8bdd-937eb5a9644a" (UID: "7878c52d-f84f-4d12-8bdd-937eb5a9644a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:00.582071 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.581974 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf" pod "7878c52d-f84f-4d12-8bdd-937eb5a9644a" (UID: "7878c52d-f84f-4d12-8bdd-937eb5a9644a"). InnerVolumeSpecName "kube-api-access-t6vxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:00.582117 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.582087 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs" pod "7878c52d-f84f-4d12-8bdd-937eb5a9644a" (UID: "7878c52d-f84f-4d12-8bdd-937eb5a9644a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:00.680999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.680958 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878c52d-f84f-4d12-8bdd-937eb5a9644a-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:00.680999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.680992 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:00.680999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.681005 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878c52d-f84f-4d12-8bdd-937eb5a9644a-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:00.681270 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.681018 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vxf\" (UniqueName: \"kubernetes.io/projected/7878c52d-f84f-4d12-8bdd-937eb5a9644a-kube-api-access-t6vxf\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:00.816536 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.816443 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:00.961964 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:00.961923 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:01.248148 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.248120 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 14:12:01.287189 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287159 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287201 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287232 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287273 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287361 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287313 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287404 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287490 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287577 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy\") pod \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\" (UID: \"0512a21c-cfd6-47ec-9c02-4e2079109cc2\") " Oct 30 14:12:01.287867 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287631 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:01.287867 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287653 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:01.287867 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287735 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:01.287867 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.287856 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:01.288112 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.288053 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:01.288228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.288204 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.288228 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.288228 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.288465 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.288243 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.288465 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.288257 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.289967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.289936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:12:01.289967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.289947 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "kube-api-access-8bqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:01.290220 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.290200 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:01.290220 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.290211 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token" pod "0512a21c-cfd6-47ec-9c02-4e2079109cc2" (UID: "0512a21c-cfd6-47ec-9c02-4e2079109cc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:01.389339 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.389245 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bqwp\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-kube-api-access-8bqwp\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.389339 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.389281 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.389339 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.389290 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.389339 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.389298 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.389339 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.389306 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0512a21c-cfd6-47ec-9c02-4e2079109cc2-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:01.440730 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.440691 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:01.453394 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.453352 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" event={"ID":"7878c52d-f84f-4d12-8bdd-937eb5a9644a","Type":"ContainerDied","Data":"93e439d8627ba133cac8aa949fa0b3b8fd9a191116d6b960fa29c4e8e9a6f3b5"} Oct 30 14:12:01.453642 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.453472 5693 scope.go:122] "RemoveContainer" containerID="486d52d6f06843d4364aa99adb8611b84a30d2a85aaa8cded0767989f1c65d33" Oct 30 14:12:01.453734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.453693 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7" Oct 30 14:12:01.460224 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.460187 5693 generic.go:356] "Generic (PLEG): container finished" podID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerID="7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4" exitCode=0 Oct 30 14:12:01.460365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.460292 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" event={"ID":"0512a21c-cfd6-47ec-9c02-4e2079109cc2","Type":"ContainerDied","Data":"7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4"} Oct 30 14:12:01.460365 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.460349 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" event={"ID":"0512a21c-cfd6-47ec-9c02-4e2079109cc2","Type":"ContainerDied","Data":"3554027dc45e7c4d9b1a50203af833996b76acc5ccc05d861555d921ffa18c43"} Oct 30 14:12:01.460472 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.460462 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp" Oct 30 14:12:01.470917 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.470889 5693 scope.go:122] "RemoveContainer" containerID="7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4" Oct 30 14:12:01.480680 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.480660 5693 scope.go:122] "RemoveContainer" containerID="3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131" Oct 30 14:12:01.489229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.489207 5693 scope.go:122] "RemoveContainer" containerID="7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4" Oct 30 14:12:01.489486 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:01.489462 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4\": container with ID starting with 7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4 not found: ID does not exist" containerID="7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4" Oct 30 14:12:01.489586 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.489493 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4"} err="failed to get container status \"7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4\": rpc error: code = NotFound desc = could not find container \"7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4\": container with ID starting with 7e6bd0bf888d5b0dc40d1fa8f0bb186cdad1bf8dd8041148513935d2666334d4 not found: ID does not exist" Oct 30 14:12:01.489586 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.489512 5693 scope.go:122] "RemoveContainer" containerID="3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131" Oct 30 14:12:01.489851 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:01.489762 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131\": container with ID starting with 3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131 not found: ID does not exist" containerID="3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131" Oct 30 14:12:01.489851 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.489784 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131"} err="failed to get container status \"3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131\": rpc error: code = NotFound desc = could not find container \"3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131\": container with ID starting with 3f797a598616c1b40ad7396bcaa7d9376cf02bf3e401ff3d3d82147ab09c1131 not found: ID does not exist" Oct 30 14:12:01.522383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.522349 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp"] Oct 30 14:12:01.527949 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.527918 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cad2c6-openshift-default-68b69f449c-p8pkp"] Oct 30 14:12:01.560080 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.560046 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7"] Oct 30 14:12:01.563691 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:01.563659 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-51dc9a-55b7779d8b-tclp7"] Oct 30 14:12:02.105077 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:02.105025 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerName="payload-pre-processing" containerID="cri-o://cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d" gracePeriod=25 Oct 30 14:12:02.254255 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:02.254220 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" path="/var/lib/kubelet/pods/0512a21c-cfd6-47ec-9c02-4e2079109cc2/volumes" Oct 30 14:12:02.254730 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:02.254714 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7878c52d-f84f-4d12-8bdd-937eb5a9644a" path="/var/lib/kubelet/pods/7878c52d-f84f-4d12-8bdd-937eb5a9644a/volumes" Oct 30 14:12:02.703086 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:02.702993 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:03.294351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:03.294316 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:04.404918 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:04.404876 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:06.122612 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:06.122561 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:07.134742 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.134713 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:07.275668 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.275630 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:07.317906 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.317868 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:07.318107 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.318050 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:07.714291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.714252 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:07.761422 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.761397 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 14:12:07.764994 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.764974 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 14:12:07.869239 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.869205 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume\") pod \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " Oct 30 14:12:07.869438 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.869283 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume\") pod \"5156f402-98f2-4349-9956-4874b8b04ed4\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " Oct 30 14:12:07.869520 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.869474 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8\" (UniqueName: \"kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8\") pod \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\" (UID: \"a29f0d1c-172c-4b0b-9de5-29bcc6a93818\") " Oct 30 14:12:07.869581 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.869539 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7\" (UniqueName: \"kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7\") pod \"5156f402-98f2-4349-9956-4874b8b04ed4\" (UID: \"5156f402-98f2-4349-9956-4874b8b04ed4\") " Oct 30 14:12:07.869667 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.869620 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume" pod "a29f0d1c-172c-4b0b-9de5-29bcc6a93818" (UID: "a29f0d1c-172c-4b0b-9de5-29bcc6a93818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:07.869833 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.869682 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume" pod "5156f402-98f2-4349-9956-4874b8b04ed4" (UID: "5156f402-98f2-4349-9956-4874b8b04ed4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:07.870217 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.870191 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5156f402-98f2-4349-9956-4874b8b04ed4-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:07.870335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.870220 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:07.871875 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.871847 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8" pod "a29f0d1c-172c-4b0b-9de5-29bcc6a93818" (UID: "a29f0d1c-172c-4b0b-9de5-29bcc6a93818"). InnerVolumeSpecName "kube-api-access-xbqg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:07.871875 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.871866 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7" pod "5156f402-98f2-4349-9956-4874b8b04ed4" (UID: "5156f402-98f2-4349-9956-4874b8b04ed4"). InnerVolumeSpecName "kube-api-access-fr8j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:07.971266 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.971191 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqg8\" (UniqueName: \"kubernetes.io/projected/a29f0d1c-172c-4b0b-9de5-29bcc6a93818-kube-api-access-xbqg8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:07.971266 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:07.971220 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8j7\" (UniqueName: \"kubernetes.io/projected/5156f402-98f2-4349-9956-4874b8b04ed4-kube-api-access-fr8j7\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:08.456413 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.456383 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:08.509341 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.509301 5693 generic.go:356] "Generic (PLEG): container finished" podID="5156f402-98f2-4349-9956-4874b8b04ed4" containerID="91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0" exitCode=0 Oct 30 14:12:08.509522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.509351 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" Oct 30 14:12:08.509522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.509377 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" event={"ID":"5156f402-98f2-4349-9956-4874b8b04ed4","Type":"ContainerDied","Data":"91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0"} Oct 30 14:12:08.509522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.509437 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk" event={"ID":"5156f402-98f2-4349-9956-4874b8b04ed4","Type":"ContainerDied","Data":"b0f260df082ec062e0f164bb758b98e48919915c52fb109ff1b9aadffe13da84"} Oct 30 14:12:08.509522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.509461 5693 scope.go:122] "RemoveContainer" containerID="91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0" Oct 30 14:12:08.511030 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.511004 5693 generic.go:356] "Generic (PLEG): container finished" podID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerID="dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6" exitCode=0 Oct 30 14:12:08.511258 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.511061 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" Oct 30 14:12:08.511384 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.511077 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" event={"ID":"a29f0d1c-172c-4b0b-9de5-29bcc6a93818","Type":"ContainerDied","Data":"dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6"} Oct 30 14:12:08.511384 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.511370 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm" event={"ID":"a29f0d1c-172c-4b0b-9de5-29bcc6a93818","Type":"ContainerDied","Data":"0656b749b83f7e099d5a0dc98152572dffa5f9cd881920c249214b97ff95d468"} Oct 30 14:12:08.522362 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.522339 5693 scope.go:122] "RemoveContainer" containerID="91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0" Oct 30 14:12:08.522710 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:08.522675 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0\": container with ID starting with 91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0 not found: ID does not exist" containerID="91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0" Oct 30 14:12:08.522832 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.522704 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0"} err="failed to get container status \"91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0\": rpc error: code = NotFound desc = could not find container \"91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0\": container with ID starting with 91563e15bb6affaf6f72272d72f831fb25f629fe099cf0ac25efb93e1657a1a0 not found: ID does not exist" Oct 30 14:12:08.522832 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.522727 5693 scope.go:122] "RemoveContainer" containerID="dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6" Oct 30 14:12:08.531848 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.531699 5693 scope.go:122] "RemoveContainer" containerID="dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6" Oct 30 14:12:08.531995 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:08.531971 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6\": container with ID starting with dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6 not found: ID does not exist" containerID="dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6" Oct 30 14:12:08.532042 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.532001 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6"} err="failed to get container status \"dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6\": rpc error: code = NotFound desc = could not find container \"dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6\": container with ID starting with dafe7d9599226a97b78b8ac3c3c0551f89d64bd005518966e01ff602a69815b6 not found: ID does not exist" Oct 30 14:12:08.559682 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.559660 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk"] Oct 30 14:12:08.571229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.571203 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cad2c6-776fbf85f8-5hxrk"] Oct 30 14:12:08.595759 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.595724 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm"] Oct 30 14:12:08.598422 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.598399 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cad2c6-68865b6ff9-hjljm"] Oct 30 14:12:08.602719 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.602698 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:08.795199 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:08.795101 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:09.261774 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:09.261733 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:09.404981 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:09.404948 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:10.253635 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:10.253575 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" path="/var/lib/kubelet/pods/5156f402-98f2-4349-9956-4874b8b04ed4/volumes" Oct 30 14:12:10.254048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:10.254034 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" path="/var/lib/kubelet/pods/a29f0d1c-172c-4b0b-9de5-29bcc6a93818/volumes" Oct 30 14:12:10.811427 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:10.811395 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:11.371044 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:11.371008 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:11.540423 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:11.540380 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:11.697215 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:11.697134 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:11.899811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:11.899774 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.051907 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:12.051874 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.207876 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:12.207848 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:13.294474 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:13.294439 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:13.783329 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:13.783288 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:14.184272 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:14.184186 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm"] Oct 30 14:12:14.333779 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:14.333736 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:15.414091 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:15.414056 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:17.070428 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:17.070375 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:18.700750 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:18.700716 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:19.354414 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.354379 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:19.533380 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.533342 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq"] Oct 30 14:12:19.533635 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.533588 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" containerID="cri-o://ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465" gracePeriod=30 Oct 30 14:12:19.535927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.535895 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:19.679689 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.679566 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:19.859407 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.859366 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk"] Oct 30 14:12:19.887948 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:19.887915 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k"] Oct 30 14:12:20.021512 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.021470 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.044955 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.044922 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k"] Oct 30 14:12:20.045154 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.045112 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" podUID="2a1169eb-6098-414b-a78e-4b7320f35189" containerName="maas-api" containerID="cri-o://7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e" gracePeriod=30 Oct 30 14:12:20.176702 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.176673 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.308110 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.308084 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 14:12:20.333080 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.333048 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs\") pod \"2a1169eb-6098-414b-a78e-4b7320f35189\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " Oct 30 14:12:20.333291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.333111 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd\" (UniqueName: \"kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd\") pod \"2a1169eb-6098-414b-a78e-4b7320f35189\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " Oct 30 14:12:20.333291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.333199 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca\") pod \"2a1169eb-6098-414b-a78e-4b7320f35189\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " Oct 30 14:12:20.333291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.333253 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\") pod \"2a1169eb-6098-414b-a78e-4b7320f35189\" (UID: \"2a1169eb-6098-414b-a78e-4b7320f35189\") " Oct 30 14:12:20.333684 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.333655 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca" pod "2a1169eb-6098-414b-a78e-4b7320f35189" (UID: "2a1169eb-6098-414b-a78e-4b7320f35189"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:20.333930 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.333907 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1169eb-6098-414b-a78e-4b7320f35189-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:20.335561 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.335536 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd" pod "2a1169eb-6098-414b-a78e-4b7320f35189" (UID: "2a1169eb-6098-414b-a78e-4b7320f35189"). InnerVolumeSpecName "kube-api-access-7jtjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:20.335779 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.335757 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs" pod "2a1169eb-6098-414b-a78e-4b7320f35189" (UID: "2a1169eb-6098-414b-a78e-4b7320f35189"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:20.335836 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.335826 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls" pod "2a1169eb-6098-414b-a78e-4b7320f35189" (UID: "2a1169eb-6098-414b-a78e-4b7320f35189"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:20.434975 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.434937 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:20.434975 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.434969 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtjd\" (UniqueName: \"kubernetes.io/projected/2a1169eb-6098-414b-a78e-4b7320f35189-kube-api-access-7jtjd\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:20.434975 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.434979 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1169eb-6098-414b-a78e-4b7320f35189-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:20.612218 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.612121 5693 generic.go:356] "Generic (PLEG): container finished" podID="2a1169eb-6098-414b-a78e-4b7320f35189" containerID="7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e" exitCode=0 Oct 30 14:12:20.612218 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.612172 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" Oct 30 14:12:20.612439 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.612205 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" event={"ID":"2a1169eb-6098-414b-a78e-4b7320f35189","Type":"ContainerDied","Data":"7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e"} Oct 30 14:12:20.612439 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.612265 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k" event={"ID":"2a1169eb-6098-414b-a78e-4b7320f35189","Type":"ContainerDied","Data":"44a8d4d2ca3a78f5e71ac4a4b07b4190c0eab17614d4bf57232a0fb61820af80"} Oct 30 14:12:20.612439 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.612287 5693 scope.go:122] "RemoveContainer" containerID="7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e" Oct 30 14:12:20.622673 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.622653 5693 scope.go:122] "RemoveContainer" containerID="7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e" Oct 30 14:12:20.622962 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:20.622938 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e\": container with ID starting with 7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e not found: ID does not exist" containerID="7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e" Oct 30 14:12:20.623037 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.622969 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e"} err="failed to get container status \"7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e\": rpc error: code = NotFound desc = could not find container \"7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e\": container with ID starting with 7fa1423f82d448b41efee56542153993b7195320190074cb3beda983dcaf708e not found: ID does not exist" Oct 30 14:12:20.662574 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.662534 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k"] Oct 30 14:12:20.674437 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.674404 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-844856-5f87c854c8-q9f7k"] Oct 30 14:12:20.690864 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.690837 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.892639 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:20.892507 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:21.348496 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:21.348462 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:22.255491 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.255459 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1169eb-6098-414b-a78e-4b7320f35189" path="/var/lib/kubelet/pods/2a1169eb-6098-414b-a78e-4b7320f35189/volumes" Oct 30 14:12:22.367823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.367757 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 14:12:22.457939 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.457903 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume\") pod \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " Oct 30 14:12:22.458140 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.457971 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8\" (UniqueName: \"kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8\") pod \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\" (UID: \"0b7772c4-fda4-4bf7-9cd1-a419d07aff95\") " Oct 30 14:12:22.458291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.458265 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume" pod "0b7772c4-fda4-4bf7-9cd1-a419d07aff95" (UID: "0b7772c4-fda4-4bf7-9cd1-a419d07aff95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:22.458717 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.458696 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:22.460203 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.460180 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8" pod "0b7772c4-fda4-4bf7-9cd1-a419d07aff95" (UID: "0b7772c4-fda4-4bf7-9cd1-a419d07aff95"). InnerVolumeSpecName "kube-api-access-c2qt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:22.560320 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.560284 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qt8\" (UniqueName: \"kubernetes.io/projected/0b7772c4-fda4-4bf7-9cd1-a419d07aff95-kube-api-access-c2qt8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:22.627862 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.627777 5693 generic.go:356] "Generic (PLEG): container finished" podID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerID="cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d" exitCode=0 Oct 30 14:12:22.627862 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.627823 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" Oct 30 14:12:22.628074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.627857 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" event={"ID":"0b7772c4-fda4-4bf7-9cd1-a419d07aff95","Type":"ContainerDied","Data":"cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d"} Oct 30 14:12:22.628074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.627914 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9" event={"ID":"0b7772c4-fda4-4bf7-9cd1-a419d07aff95","Type":"ContainerDied","Data":"f6c86e3d805b9823732ba6c7a0af8d653451b68896728dd9879520a9caafc5fe"} Oct 30 14:12:22.628074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.627953 5693 scope.go:122] "RemoveContainer" containerID="cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d" Oct 30 14:12:22.638023 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.638003 5693 scope.go:122] "RemoveContainer" containerID="cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d" Oct 30 14:12:22.638288 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:22.638261 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d\": container with ID starting with cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d not found: ID does not exist" containerID="cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d" Oct 30 14:12:22.638366 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.638304 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d"} err="failed to get container status \"cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d\": rpc error: code = NotFound desc = could not find container \"cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d\": container with ID starting with cce95c32bf47463028760e01bec901087454a5fedfd3f7dd27fa0bc1ac6b736d not found: ID does not exist" Oct 30 14:12:22.679850 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.679819 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9"] Oct 30 14:12:22.691159 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.691132 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f7ee8a-855fc677f9-bhwn9"] Oct 30 14:12:22.712827 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.712800 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:22.866058 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:22.866024 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:23.055734 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:23.055695 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:12:24.255975 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.255939 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" path="/var/lib/kubelet/pods/0b7772c4-fda4-4bf7-9cd1-a419d07aff95/volumes" Oct 30 14:12:24.520472 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.520378 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:24.784426 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.784366 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 14:12:24.860630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.860579 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" podUID="201a039d-07be-4f7a-b194-7a536c6fdfcc" containerName="payload-processing" containerID="cri-o://3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4" gracePeriod=25 Oct 30 14:12:24.887811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.887770 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.887951 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.887875 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.887951 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.887924 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888082 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.887987 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888082 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888005 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888084 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888202 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888140 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888200 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888279 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert\") pod \"e04aea9d-1efd-4626-9bea-8449107bf8b0\" (UID: \"e04aea9d-1efd-4626-9bea-8449107bf8b0\") " Oct 30 14:12:24.888325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888271 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:24.888490 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888372 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:24.888490 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888465 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:24.888828 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888804 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:24.888976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888862 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.888976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888888 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.888976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888867 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" podUID="a41b1373-0002-4a40-9a9d-abff679404c0" containerName="payload-pre-processing" containerID="cri-o://dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e" gracePeriod=25 Oct 30 14:12:24.888976 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.888906 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.889629 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.889197 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:24.890502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.890476 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:24.890588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.890530 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "kube-api-access-pdxx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:24.890758 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.890728 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:12:24.890758 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.890732 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy" pod "e04aea9d-1efd-4626-9bea-8449107bf8b0" (UID: "e04aea9d-1efd-4626-9bea-8449107bf8b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:24.989872 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.989841 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.989872 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.989871 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdxx8\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-kube-api-access-pdxx8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.989872 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.989883 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04aea9d-1efd-4626-9bea-8449107bf8b0-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.990088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.989893 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.990088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.989902 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04aea9d-1efd-4626-9bea-8449107bf8b0-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:24.990088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:24.989912 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04aea9d-1efd-4626-9bea-8449107bf8b0-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:25.139740 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.139704 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 14:12:25.178818 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.178790 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 14:12:25.192656 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.192621 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume\") pod \"201a039d-07be-4f7a-b194-7a536c6fdfcc\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " Oct 30 14:12:25.192837 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.192755 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9\" (UniqueName: \"kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9\") pod \"201a039d-07be-4f7a-b194-7a536c6fdfcc\" (UID: \"201a039d-07be-4f7a-b194-7a536c6fdfcc\") " Oct 30 14:12:25.192987 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.192960 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume" pod "201a039d-07be-4f7a-b194-7a536c6fdfcc" (UID: "201a039d-07be-4f7a-b194-7a536c6fdfcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:25.193151 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.193134 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201a039d-07be-4f7a-b194-7a536c6fdfcc-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:25.195389 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.195365 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9" pod "201a039d-07be-4f7a-b194-7a536c6fdfcc" (UID: "201a039d-07be-4f7a-b194-7a536c6fdfcc"). InnerVolumeSpecName "kube-api-access-mzlb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:25.294079 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.294045 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume\") pod \"a41b1373-0002-4a40-9a9d-abff679404c0\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " Oct 30 14:12:25.294545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.294276 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9\" (UniqueName: \"kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9\") pod \"a41b1373-0002-4a40-9a9d-abff679404c0\" (UID: \"a41b1373-0002-4a40-9a9d-abff679404c0\") " Oct 30 14:12:25.294545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.294411 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume" pod "a41b1373-0002-4a40-9a9d-abff679404c0" (UID: "a41b1373-0002-4a40-9a9d-abff679404c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:25.294683 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.294666 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlb9\" (UniqueName: \"kubernetes.io/projected/201a039d-07be-4f7a-b194-7a536c6fdfcc-kube-api-access-mzlb9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:25.294727 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.294685 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41b1373-0002-4a40-9a9d-abff679404c0-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:25.296367 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.296343 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9" pod "a41b1373-0002-4a40-9a9d-abff679404c0" (UID: "a41b1373-0002-4a40-9a9d-abff679404c0"). InnerVolumeSpecName "kube-api-access-jl5x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:25.395869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.395784 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl5x9\" (UniqueName: \"kubernetes.io/projected/a41b1373-0002-4a40-9a9d-abff679404c0-kube-api-access-jl5x9\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:25.648307 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.648214 5693 generic.go:356] "Generic (PLEG): container finished" podID="201a039d-07be-4f7a-b194-7a536c6fdfcc" containerID="3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4" exitCode=0 Oct 30 14:12:25.648307 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.648266 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" Oct 30 14:12:25.648307 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.648294 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" event={"ID":"201a039d-07be-4f7a-b194-7a536c6fdfcc","Type":"ContainerDied","Data":"3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4"} Oct 30 14:12:25.648775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.648352 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk" event={"ID":"201a039d-07be-4f7a-b194-7a536c6fdfcc","Type":"ContainerDied","Data":"9c63761b9f6aa65cc53816727b1b8e1304a15e1f4bff5a906e4b7c924d3c3f23"} Oct 30 14:12:25.648775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.648375 5693 scope.go:122] "RemoveContainer" containerID="3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4" Oct 30 14:12:25.650311 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.650289 5693 generic.go:356] "Generic (PLEG): container finished" podID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerID="ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465" exitCode=0 Oct 30 14:12:25.650448 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.650353 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" Oct 30 14:12:25.650448 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.650360 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" event={"ID":"e04aea9d-1efd-4626-9bea-8449107bf8b0","Type":"ContainerDied","Data":"ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465"} Oct 30 14:12:25.650448 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.650401 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq" event={"ID":"e04aea9d-1efd-4626-9bea-8449107bf8b0","Type":"ContainerDied","Data":"9b043ae455911d73a3c99d0e9f90a8705ec5fbf0dc1baa8173674007601c676e"} Oct 30 14:12:25.651913 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.651892 5693 generic.go:356] "Generic (PLEG): container finished" podID="a41b1373-0002-4a40-9a9d-abff679404c0" containerID="dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e" exitCode=0 Oct 30 14:12:25.652065 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.652050 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" event={"ID":"a41b1373-0002-4a40-9a9d-abff679404c0","Type":"ContainerDied","Data":"dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e"} Oct 30 14:12:25.652139 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.652082 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" event={"ID":"a41b1373-0002-4a40-9a9d-abff679404c0","Type":"ContainerDied","Data":"c642fe12b440de70f8c71fdbe57458938406f3280e9c5465301acc6acc94ba1e"} Oct 30 14:12:25.652139 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.652130 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k" Oct 30 14:12:25.661126 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.661098 5693 scope.go:122] "RemoveContainer" containerID="3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4" Oct 30 14:12:25.661460 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:25.661429 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4\": container with ID starting with 3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4 not found: ID does not exist" containerID="3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4" Oct 30 14:12:25.661565 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.661469 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4"} err="failed to get container status \"3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4\": rpc error: code = NotFound desc = could not find container \"3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4\": container with ID starting with 3dfa565e3e26f95010d0179cc792184162bf47dd347f4166236ea3e3309de2a4 not found: ID does not exist" Oct 30 14:12:25.661565 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.661496 5693 scope.go:122] "RemoveContainer" containerID="ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465" Oct 30 14:12:25.672546 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.672527 5693 scope.go:122] "RemoveContainer" containerID="6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a" Oct 30 14:12:25.681240 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.681222 5693 scope.go:122] "RemoveContainer" containerID="ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465" Oct 30 14:12:25.681478 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:25.681459 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465\": container with ID starting with ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465 not found: ID does not exist" containerID="ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465" Oct 30 14:12:25.681560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.681489 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465"} err="failed to get container status \"ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465\": rpc error: code = NotFound desc = could not find container \"ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465\": container with ID starting with ba0263535cf16d258128994b5ff57ac38d268f6a2daeb9e47a36a3ae97d22465 not found: ID does not exist" Oct 30 14:12:25.681560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.681512 5693 scope.go:122] "RemoveContainer" containerID="6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a" Oct 30 14:12:25.681777 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:25.681758 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a\": container with ID starting with 6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a not found: ID does not exist" containerID="6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a" Oct 30 14:12:25.681824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.681784 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a"} err="failed to get container status \"6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a\": rpc error: code = NotFound desc = could not find container \"6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a\": container with ID starting with 6d56f30836178e1107f4487266102b4992d8eee398bc31277e9996b503ee231a not found: ID does not exist" Oct 30 14:12:25.681824 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.681800 5693 scope.go:122] "RemoveContainer" containerID="dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e" Oct 30 14:12:25.689900 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.689881 5693 scope.go:122] "RemoveContainer" containerID="dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e" Oct 30 14:12:25.690177 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:25.690158 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e\": container with ID starting with dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e not found: ID does not exist" containerID="dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e" Oct 30 14:12:25.690243 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.690183 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e"} err="failed to get container status \"dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e\": rpc error: code = NotFound desc = could not find container \"dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e\": container with ID starting with dafacca3197eed2efc11b823039dae11f287e391d3aac71e0ac0e20e2dda6f7e not found: ID does not exist" Oct 30 14:12:25.728325 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.728298 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k"] Oct 30 14:12:25.732635 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.732582 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f9ebb5-94dbbf4dc-xhb8k"] Oct 30 14:12:25.764461 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.764435 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq"] Oct 30 14:12:25.770305 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.770282 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f7ee8a-openshift-default-5db67d7b77-pdrqq"] Oct 30 14:12:25.803523 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.803486 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk"] Oct 30 14:12:25.810020 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:25.809994 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f9ebb5-6d95768b48-wgdqk"] Oct 30 14:12:26.213304 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:26.213265 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:26.255042 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:26.255008 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201a039d-07be-4f7a-b194-7a536c6fdfcc" path="/var/lib/kubelet/pods/201a039d-07be-4f7a-b194-7a536c6fdfcc/volumes" Oct 30 14:12:26.255507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:26.255492 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41b1373-0002-4a40-9a9d-abff679404c0" path="/var/lib/kubelet/pods/a41b1373-0002-4a40-9a9d-abff679404c0/volumes" Oct 30 14:12:26.255999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:26.255982 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" path="/var/lib/kubelet/pods/e04aea9d-1efd-4626-9bea-8449107bf8b0/volumes" Oct 30 14:12:27.816501 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:27.816456 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:28.003051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:28.003004 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:28.152833 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:28.152735 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:28.341281 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:28.341251 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg"] Oct 30 14:12:28.341489 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:28.341469 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" containerID="cri-o://9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23" gracePeriod=30 Oct 30 14:12:28.413765 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:28.413664 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:29.205777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.205745 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" containerID="cri-o://1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e" gracePeriod=45 Oct 30 14:12:29.476188 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.476159 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 14:12:29.542975 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.542938 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir\") pod \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " Oct 30 14:12:29.543192 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.543012 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs\") pod \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " Oct 30 14:12:29.543192 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.543060 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk\" (UniqueName: \"kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk\") pod \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " Oct 30 14:12:29.543192 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.543083 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache\") pod \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " Oct 30 14:12:29.543192 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.543180 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home\") pod \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " Oct 30 14:12:29.543467 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.543241 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm\") pod \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\" (UID: \"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4\") " Oct 30 14:12:29.544134 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.544104 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir" pod "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" (UID: "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:29.544402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.544374 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache" pod "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" (UID: "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:29.544489 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.544463 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home" pod "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" (UID: "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:29.547085 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.547029 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs" pod "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" (UID: "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:29.547085 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.547038 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk" pod "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" (UID: "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4"). InnerVolumeSpecName "kube-api-access-cfkrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:29.547241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.547105 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm" pod "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" (UID: "6cffe0f8-7282-4e8f-ae05-d15c86ef35d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:29.644709 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.644669 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkrk\" (UniqueName: \"kubernetes.io/projected/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-kube-api-access-cfkrk\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:29.644709 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.644707 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:29.644986 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.644721 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:29.644986 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.644733 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:29.644986 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.644745 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:29.644986 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.644756 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:29.681530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.681429 5693 generic.go:356] "Generic (PLEG): container finished" podID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerID="1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e" exitCode=0 Oct 30 14:12:29.681762 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.681546 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" event={"ID":"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4","Type":"ContainerDied","Data":"1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e"} Oct 30 14:12:29.681762 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.681588 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" event={"ID":"6cffe0f8-7282-4e8f-ae05-d15c86ef35d4","Type":"ContainerDied","Data":"dbda0be3b5e3c0fbe34d31873bf28df926d48703e0aef2905880fa5b7cecd468"} Oct 30 14:12:29.681762 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.681636 5693 scope.go:122] "RemoveContainer" containerID="1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e" Oct 30 14:12:29.681762 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.681635 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm" Oct 30 14:12:29.692825 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.692802 5693 scope.go:122] "RemoveContainer" containerID="1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e" Oct 30 14:12:29.693168 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:29.693136 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e\": container with ID starting with 1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e not found: ID does not exist" containerID="1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e" Oct 30 14:12:29.693263 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.693176 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e"} err="failed to get container status \"1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e\": rpc error: code = NotFound desc = could not find container \"1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e\": container with ID starting with 1d41a81e58cb389f884a2806c52aa97f28085135a92bd70150ca17e6b4cca49e not found: ID does not exist" Oct 30 14:12:29.730978 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.730953 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm"] Oct 30 14:12:29.735116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.735092 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-2-w0-f9ebb5-kserve-856f8c577-rq4sm"] Oct 30 14:12:29.782945 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:29.782921 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.254148 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:30.254107 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" path="/var/lib/kubelet/pods/6cffe0f8-7282-4e8f-ae05-d15c86ef35d4/volumes" Oct 30 14:12:30.687018 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:30.686913 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.832772 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:30.832724 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.905965 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:30.905923 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:12:30.906128 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:30.905991 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:12:31.504165 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:31.504128 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:33.211039 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.210990 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:33.599093 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.599071 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 14:12:33.696836 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.696797 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697009 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.696863 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697009 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.696926 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697009 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.696969 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697211 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697038 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697211 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697070 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697211 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697100 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697211 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697170 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:33.697211 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697180 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697486 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697325 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:33.697486 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697333 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo\") pod \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\" (UID: \"2a9623ae-3597-4add-97e7-a21f3b38c6b6\") " Oct 30 14:12:33.697486 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697362 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:33.697486 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697381 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:33.697741 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697727 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:33.697926 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697904 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.698007 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697930 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.698007 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697945 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.698007 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697960 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.698007 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.697973 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.699530 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.699503 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:33.699735 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.699525 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:12:33.699933 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.699908 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:33.700250 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.700225 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg" pod "2a9623ae-3597-4add-97e7-a21f3b38c6b6" (UID: "2a9623ae-3597-4add-97e7-a21f3b38c6b6"). InnerVolumeSpecName "kube-api-access-79xcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:33.710040 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.710011 5693 generic.go:356] "Generic (PLEG): container finished" podID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerID="9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23" exitCode=0 Oct 30 14:12:33.710164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.710065 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" Oct 30 14:12:33.710164 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.710086 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" event={"ID":"2a9623ae-3597-4add-97e7-a21f3b38c6b6","Type":"ContainerDied","Data":"9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23"} Oct 30 14:12:33.710282 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.710185 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg" event={"ID":"2a9623ae-3597-4add-97e7-a21f3b38c6b6","Type":"ContainerDied","Data":"50bc178d6a2519b6749589f2757944b096b3c2478c122e2f91c4f8d2b9a86f31"} Oct 30 14:12:33.710282 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.710219 5693 scope.go:122] "RemoveContainer" containerID="9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23" Oct 30 14:12:33.727827 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.727808 5693 scope.go:122] "RemoveContainer" containerID="c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2" Oct 30 14:12:33.738693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.738673 5693 scope.go:122] "RemoveContainer" containerID="9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23" Oct 30 14:12:33.739026 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:33.739000 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23\": container with ID starting with 9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23 not found: ID does not exist" containerID="9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23" Oct 30 14:12:33.739118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.739036 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23"} err="failed to get container status \"9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23\": rpc error: code = NotFound desc = could not find container \"9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23\": container with ID starting with 9949ce18bc5bb85be75f0e692ab6f7614fa8711939746f79cbf13688ab392a23 not found: ID does not exist" Oct 30 14:12:33.739118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.739052 5693 scope.go:122] "RemoveContainer" containerID="c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2" Oct 30 14:12:33.739339 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:33.739315 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2\": container with ID starting with c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2 not found: ID does not exist" containerID="c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2" Oct 30 14:12:33.739407 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.739343 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2"} err="failed to get container status \"c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2\": rpc error: code = NotFound desc = could not find container \"c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2\": container with ID starting with c95128ae2cda2b8873d8157031de7c7d33bbd72499f5b127c88f9bda552d07c2 not found: ID does not exist" Oct 30 14:12:33.760073 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.759941 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg"] Oct 30 14:12:33.764823 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.764759 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-chbhg"] Oct 30 14:12:33.799223 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.799195 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.799223 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.799218 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.799223 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.799227 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9623ae-3597-4add-97e7-a21f3b38c6b6-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:33.799402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:33.799236 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79xcg\" (UniqueName: \"kubernetes.io/projected/2a9623ae-3597-4add-97e7-a21f3b38c6b6-kube-api-access-79xcg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:34.140106 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:34.140009 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:34.254776 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:34.254745 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" path="/var/lib/kubelet/pods/2a9623ae-3597-4add-97e7-a21f3b38c6b6/volumes" Oct 30 14:12:34.297208 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:34.297173 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:34.906304 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:34.906267 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:35.423241 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:35.423206 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj"] Oct 30 14:12:35.423701 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:35.423453 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" containerID="cri-o://a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa" gracePeriod=30 Oct 30 14:12:36.604083 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:36.604041 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:37.985943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:37.985901 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768"] Oct 30 14:12:37.986435 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:37.986320 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" podUID="d1b5a289-4aea-4a69-9544-a3326448333a" containerName="maas-api" containerID="cri-o://d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a" gracePeriod=30 Oct 30 14:12:38.088425 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.088389 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:38.257004 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.256977 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 14:12:38.349413 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.349382 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs\") pod \"d1b5a289-4aea-4a69-9544-a3326448333a\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " Oct 30 14:12:38.349632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.349435 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls\") pod \"d1b5a289-4aea-4a69-9544-a3326448333a\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " Oct 30 14:12:38.349632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.349475 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca\") pod \"d1b5a289-4aea-4a69-9544-a3326448333a\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " Oct 30 14:12:38.349632 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.349559 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm\" (UniqueName: \"kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm\") pod \"d1b5a289-4aea-4a69-9544-a3326448333a\" (UID: \"d1b5a289-4aea-4a69-9544-a3326448333a\") " Oct 30 14:12:38.349969 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.349942 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca" pod "d1b5a289-4aea-4a69-9544-a3326448333a" (UID: "d1b5a289-4aea-4a69-9544-a3326448333a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:38.350084 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.350067 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b5a289-4aea-4a69-9544-a3326448333a-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:38.351526 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.351490 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs" pod "d1b5a289-4aea-4a69-9544-a3326448333a" (UID: "d1b5a289-4aea-4a69-9544-a3326448333a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:38.351696 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.351549 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls" pod "d1b5a289-4aea-4a69-9544-a3326448333a" (UID: "d1b5a289-4aea-4a69-9544-a3326448333a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:38.351696 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.351674 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm" pod "d1b5a289-4aea-4a69-9544-a3326448333a" (UID: "d1b5a289-4aea-4a69-9544-a3326448333a"). InnerVolumeSpecName "kube-api-access-486fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:38.451275 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.451237 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:38.451275 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.451271 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b5a289-4aea-4a69-9544-a3326448333a-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:38.451471 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.451283 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486fm\" (UniqueName: \"kubernetes.io/projected/d1b5a289-4aea-4a69-9544-a3326448333a-kube-api-access-486fm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:38.598345 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.598266 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:38.743206 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.743169 5693 generic.go:356] "Generic (PLEG): container finished" podID="d1b5a289-4aea-4a69-9544-a3326448333a" containerID="d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a" exitCode=0 Oct 30 14:12:38.743387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.743219 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" Oct 30 14:12:38.743387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.743250 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" event={"ID":"d1b5a289-4aea-4a69-9544-a3326448333a","Type":"ContainerDied","Data":"d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a"} Oct 30 14:12:38.743387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.743302 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768" event={"ID":"d1b5a289-4aea-4a69-9544-a3326448333a","Type":"ContainerDied","Data":"f048ffc74cf181991c954ecbd6c24c81f50ebb70ef4ce92f30cb3eca56d283d7"} Oct 30 14:12:38.743387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.743324 5693 scope.go:122] "RemoveContainer" containerID="d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a" Oct 30 14:12:38.753199 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.753183 5693 scope.go:122] "RemoveContainer" containerID="d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a" Oct 30 14:12:38.753457 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:38.753438 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a\": container with ID starting with d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a not found: ID does not exist" containerID="d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a" Oct 30 14:12:38.753504 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.753468 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a"} err="failed to get container status \"d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a\": rpc error: code = NotFound desc = could not find container \"d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a\": container with ID starting with d27ddb416f98b873480b7372cf37acad7a6afb1779e863a72a358d16afadd80a not found: ID does not exist" Oct 30 14:12:38.759591 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.759551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:38.796198 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.796173 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768"] Oct 30 14:12:38.807983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:38.807958 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-29a3f3-f5fb6cd44-7r768"] Oct 30 14:12:39.242835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:39.242800 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct"] Oct 30 14:12:39.256934 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:39.256886 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:39.273432 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:39.273406 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw"] Oct 30 14:12:39.415257 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:39.415221 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:39.555274 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:39.555176 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:40.253774 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.253731 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b5a289-4aea-4a69-9544-a3326448333a" path="/var/lib/kubelet/pods/d1b5a289-4aea-4a69-9544-a3326448333a/volumes" Oct 30 14:12:40.499142 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.499107 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:40.670652 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.670625 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:12:40.757377 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.757335 5693 generic.go:356] "Generic (PLEG): container finished" podID="1ffcbe21-629f-402e-9847-02108c1a0392" containerID="a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa" exitCode=0 Oct 30 14:12:40.757575 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.757378 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" Oct 30 14:12:40.757575 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.757406 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" event={"ID":"1ffcbe21-629f-402e-9847-02108c1a0392","Type":"ContainerDied","Data":"a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa"} Oct 30 14:12:40.757575 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.757460 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj" event={"ID":"1ffcbe21-629f-402e-9847-02108c1a0392","Type":"ContainerDied","Data":"f203bfc46a2714815cfc380b4c88a5680bf4ee89ac9ef720c811c978e4e811b6"} Oct 30 14:12:40.757575 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.757483 5693 scope.go:122] "RemoveContainer" containerID="a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa" Oct 30 14:12:40.769567 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.769547 5693 scope.go:122] "RemoveContainer" containerID="a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa" Oct 30 14:12:40.769884 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:40.769856 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa\": container with ID starting with a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa not found: ID does not exist" containerID="a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa" Oct 30 14:12:40.769970 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.769896 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa"} err="failed to get container status \"a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa\": rpc error: code = NotFound desc = could not find container \"a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa\": container with ID starting with a252279cdfbcc21c14c94cdd4814d8bfd2067485878b778fb556f30d21e190fa not found: ID does not exist" Oct 30 14:12:40.775836 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.775815 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.775932 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.775886 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.775993 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.775932 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.775993 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.775978 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.776078 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776005 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.776078 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776056 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.776162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776073 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:40.776162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776116 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.776162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776143 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:40.776292 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776159 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.776292 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776222 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5\") pod \"1ffcbe21-629f-402e-9847-02108c1a0392\" (UID: \"1ffcbe21-629f-402e-9847-02108c1a0392\") " Oct 30 14:12:40.776404 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776322 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:40.776522 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776495 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:40.776709 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776508 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:40.776786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776743 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffcbe21-629f-402e-9847-02108c1a0392-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.776786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776764 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.776786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776778 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.776960 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776792 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.776960 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.776807 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.778134 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.778109 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:12:40.778245 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.778171 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:40.778245 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.778199 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:40.778519 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.778499 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5" pod "1ffcbe21-629f-402e-9847-02108c1a0392" (UID: "1ffcbe21-629f-402e-9847-02108c1a0392"). InnerVolumeSpecName "kube-api-access-dstx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:40.878516 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.878432 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffcbe21-629f-402e-9847-02108c1a0392-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.878516 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.878465 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstx5\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-kube-api-access-dstx5\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.878516 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.878475 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffcbe21-629f-402e-9847-02108c1a0392-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:40.878516 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:40.878484 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffcbe21-629f-402e-9847-02108c1a0392-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:41.101957 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:41.101920 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj"] Oct 30 14:12:41.113461 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:41.113427 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5w2dj"] Oct 30 14:12:42.104050 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:42.104010 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:42.253999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:42.253965 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" path="/var/lib/kubelet/pods/1ffcbe21-629f-402e-9847-02108c1a0392/volumes" Oct 30 14:12:43.527663 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:43.527625 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:44.007529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:44.007499 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:44.152510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:44.152471 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:44.243999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:44.243956 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" containerName="payload-processing" containerID="cri-o://3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8" gracePeriod=25 Oct 30 14:12:44.274775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:44.274706 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" podUID="7022b194-fe9d-4290-832b-4380c59eec72" containerName="payload-pre-processing" containerID="cri-o://81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf" gracePeriod=25 Oct 30 14:12:44.620350 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:44.620262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:45.406177 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.406143 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d"] Oct 30 14:12:45.406411 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.406373 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" podUID="1a3e16f5-7af6-428a-a80e-fc16c17b6f51" containerName="maas-api" containerID="cri-o://1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76" gracePeriod=30 Oct 30 14:12:45.507892 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.507850 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:45.670142 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.670071 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 14:12:45.788505 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.788469 5693 generic.go:356] "Generic (PLEG): container finished" podID="1a3e16f5-7af6-428a-a80e-fc16c17b6f51" containerID="1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76" exitCode=0 Oct 30 14:12:45.788734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.788512 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" Oct 30 14:12:45.788734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.788533 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" event={"ID":"1a3e16f5-7af6-428a-a80e-fc16c17b6f51","Type":"ContainerDied","Data":"1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76"} Oct 30 14:12:45.788734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.788572 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d" event={"ID":"1a3e16f5-7af6-428a-a80e-fc16c17b6f51","Type":"ContainerDied","Data":"bf53e50e2f656ddd71214572092d92202a2184adac595b2f28c4f200c2da3028"} Oct 30 14:12:45.788734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.788593 5693 scope.go:122] "RemoveContainer" containerID="1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76" Oct 30 14:12:45.797927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.797902 5693 scope.go:122] "RemoveContainer" containerID="1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76" Oct 30 14:12:45.798243 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:45.798217 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76\": container with ID starting with 1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76 not found: ID does not exist" containerID="1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76" Oct 30 14:12:45.798319 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.798258 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76"} err="failed to get container status \"1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76\": rpc error: code = NotFound desc = could not find container \"1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76\": container with ID starting with 1031097ea98f1553cf8a8522e0cf8334d6346003d9a4b4eb09a16e59ae28bc76 not found: ID does not exist" Oct 30 14:12:45.829892 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.829864 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca\") pod \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " Oct 30 14:12:45.830077 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.829956 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs\") pod \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " Oct 30 14:12:45.830077 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.829981 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls\") pod \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " Oct 30 14:12:45.830077 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.830044 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg\" (UniqueName: \"kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg\") pod \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\" (UID: \"1a3e16f5-7af6-428a-a80e-fc16c17b6f51\") " Oct 30 14:12:45.830326 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.830293 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca" pod "1a3e16f5-7af6-428a-a80e-fc16c17b6f51" (UID: "1a3e16f5-7af6-428a-a80e-fc16c17b6f51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:45.830533 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.830509 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:45.832264 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.832234 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs" pod "1a3e16f5-7af6-428a-a80e-fc16c17b6f51" (UID: "1a3e16f5-7af6-428a-a80e-fc16c17b6f51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:45.832358 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.832279 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg" pod "1a3e16f5-7af6-428a-a80e-fc16c17b6f51" (UID: "1a3e16f5-7af6-428a-a80e-fc16c17b6f51"). InnerVolumeSpecName "kube-api-access-bkwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:45.832541 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.832516 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls" pod "1a3e16f5-7af6-428a-a80e-fc16c17b6f51" (UID: "1a3e16f5-7af6-428a-a80e-fc16c17b6f51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:45.932333 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.932239 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:45.932333 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.932283 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:45.932333 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:45.932297 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwxg\" (UniqueName: \"kubernetes.io/projected/1a3e16f5-7af6-428a-a80e-fc16c17b6f51-kube-api-access-bkwxg\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:46.008696 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.008661 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:46.132638 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.132581 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d"] Oct 30 14:12:46.136573 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.136543 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f9ebb5-595bb4c74d-q948d"] Oct 30 14:12:46.152925 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.152897 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:46.225127 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.225091 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:46.225127 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.225111 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" podUID="7022b194-fe9d-4290-832b-4380c59eec72" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:46.253721 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.253689 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3e16f5-7af6-428a-a80e-fc16c17b6f51" path="/var/lib/kubelet/pods/1a3e16f5-7af6-428a-a80e-fc16c17b6f51/volumes" Oct 30 14:12:46.611657 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:46.611546 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:47.173783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:47.173746 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:47.338351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:47.338313 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:47.627287 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:47.627249 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:47.918108 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:47.917998 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:48.065173 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:48.065140 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:49.708672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:49.708618 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:50.550869 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:50.550841 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:50.701724 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:50.701644 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:51.438465 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:51.438436 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:53.057554 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:53.057511 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:12:53.065963 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:53.065933 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p"] Oct 30 14:12:53.189294 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:53.189262 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:53.357054 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:53.356983 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:53.512914 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:53.512873 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9"] Oct 30 14:12:53.513219 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:53.513191 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" containerID="cri-o://575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949" gracePeriod=30 Oct 30 14:12:53.632034 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:53.631934 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:54.473588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:54.473547 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:56.203170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:56.203137 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:56.225461 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:56.225424 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" podUID="7022b194-fe9d-4290-832b-4380c59eec72" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:56.225666 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:56.225433 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:12:56.354954 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:56.354898 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:56.826354 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:56.826307 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:57.286433 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:57.286397 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:12:57.286814 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:57.286448 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:12:58.407284 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.407242 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:58.612233 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.612202 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww"] Oct 30 14:12:58.645996 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.645963 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f"] Oct 30 14:12:58.785905 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.785873 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:58.796497 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.796470 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:12:58.866532 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.866500 5693 generic.go:356] "Generic (PLEG): container finished" podID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerID="575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949" exitCode=0 Oct 30 14:12:58.866734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.866566 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" Oct 30 14:12:58.866734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.866577 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" event={"ID":"a394b55a-96df-4246-86b1-bd9d3f98eb18","Type":"ContainerDied","Data":"575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949"} Oct 30 14:12:58.866734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.866666 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9" event={"ID":"a394b55a-96df-4246-86b1-bd9d3f98eb18","Type":"ContainerDied","Data":"e97bb85b537f7890b92009a7b8e54e0ddf94a83df7bdc393eb15fd834e3481f8"} Oct 30 14:12:58.866734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.866695 5693 scope.go:122] "RemoveContainer" containerID="575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949" Oct 30 14:12:58.876918 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.876893 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877068 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.876938 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877068 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.876973 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877068 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877061 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877085 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877138 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877230 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877249 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:58.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877264 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877341 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo\") pod \"a394b55a-96df-4246-86b1-bd9d3f98eb18\" (UID: \"a394b55a-96df-4246-86b1-bd9d3f98eb18\") " Oct 30 14:12:58.877747 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877461 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:58.877747 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877460 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:58.877747 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877712 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:58.877747 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877726 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:58.877940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877866 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a394b55a-96df-4246-86b1-bd9d3f98eb18-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.877940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877884 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.877940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877897 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.877940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877910 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.877940 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.877922 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.879054 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.879026 5693 scope.go:122] "RemoveContainer" containerID="575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949" Oct 30 14:12:58.879592 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:12:58.879545 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949\": container with ID starting with 575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949 not found: ID does not exist" containerID="575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949" Oct 30 14:12:58.879592 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.879574 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949"} err="failed to get container status \"575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949\": rpc error: code = NotFound desc = could not find container \"575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949\": container with ID starting with 575173a0c78e5bbdba8e5dd74829ae4fd232def8d6d22e585fc7c68673e49949 not found: ID does not exist" Oct 30 14:12:58.879781 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.879720 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "kube-api-access-lz2gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:58.879834 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.879814 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:58.880477 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.880449 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:58.880850 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.880824 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo" pod "a394b55a-96df-4246-86b1-bd9d3f98eb18" (UID: "a394b55a-96df-4246-86b1-bd9d3f98eb18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:12:58.949128 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.949053 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:58.979417 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.979381 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.979417 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.979413 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.979644 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.979425 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz2gb\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-kube-api-access-lz2gb\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:58.979644 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:58.979443 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a394b55a-96df-4246-86b1-bd9d3f98eb18-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:12:59.215274 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:59.215182 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9"] Oct 30 14:12:59.229404 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:59.229357 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-pb8r9"] Oct 30 14:12:59.632439 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:12:59.632399 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:00.127947 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:00.127906 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:00.254261 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:00.254228 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" path="/var/lib/kubelet/pods/a394b55a-96df-4246-86b1-bd9d3f98eb18/volumes" Oct 30 14:13:00.738861 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:00.738817 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:01.783805 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:01.783769 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:03.419544 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.419516 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:03.612812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.612775 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" podUID="b571d1ac-9679-4f13-957e-f87bf892c875" containerName="payload-processing" containerID="cri-o://6a256bf9d3b8c8563ba30bbd635df26a75f21168c8ad4edf11d69c56e799a9e7" gracePeriod=25 Oct 30 14:13:03.646429 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.646381 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" podUID="fbdb9cc2-807a-437e-96c7-454f224e067e" containerName="payload-pre-processing" containerID="cri-o://744ca13c4555f220af2005cca159f4e3a582a45a19392f48734e4c4fb6f07403" gracePeriod=25 Oct 30 14:13:03.899038 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.899008 5693 generic.go:356] "Generic (PLEG): container finished" podID="fbdb9cc2-807a-437e-96c7-454f224e067e" containerID="744ca13c4555f220af2005cca159f4e3a582a45a19392f48734e4c4fb6f07403" exitCode=0 Oct 30 14:13:03.899196 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.899085 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" event={"ID":"fbdb9cc2-807a-437e-96c7-454f224e067e","Type":"ContainerDied","Data":"744ca13c4555f220af2005cca159f4e3a582a45a19392f48734e4c4fb6f07403"} Oct 30 14:13:03.900680 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.900658 5693 generic.go:356] "Generic (PLEG): container finished" podID="b571d1ac-9679-4f13-957e-f87bf892c875" containerID="6a256bf9d3b8c8563ba30bbd635df26a75f21168c8ad4edf11d69c56e799a9e7" exitCode=0 Oct 30 14:13:03.900811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.900686 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" event={"ID":"b571d1ac-9679-4f13-957e-f87bf892c875","Type":"ContainerDied","Data":"6a256bf9d3b8c8563ba30bbd635df26a75f21168c8ad4edf11d69c56e799a9e7"} Oct 30 14:13:03.900811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.900722 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" event={"ID":"b571d1ac-9679-4f13-957e-f87bf892c875","Type":"ContainerDied","Data":"12aab7e0b6dfdc270c560a27ccde1d3dba6ee1d0d6c847e0b8377e533cbce13b"} Oct 30 14:13:03.900811 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.900738 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12aab7e0b6dfdc270c560a27ccde1d3dba6ee1d0d6c847e0b8377e533cbce13b" Oct 30 14:13:03.903491 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.903473 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 14:13:03.932352 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:03.932329 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 14:13:04.032698 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.032669 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume\") pod \"b571d1ac-9679-4f13-957e-f87bf892c875\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " Oct 30 14:13:04.032893 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.032746 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2\" (UniqueName: \"kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2\") pod \"fbdb9cc2-807a-437e-96c7-454f224e067e\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " Oct 30 14:13:04.032893 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.032780 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume\") pod \"fbdb9cc2-807a-437e-96c7-454f224e067e\" (UID: \"fbdb9cc2-807a-437e-96c7-454f224e067e\") " Oct 30 14:13:04.032893 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.032808 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l\" (UniqueName: \"kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l\") pod \"b571d1ac-9679-4f13-957e-f87bf892c875\" (UID: \"b571d1ac-9679-4f13-957e-f87bf892c875\") " Oct 30 14:13:04.033087 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.033042 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume" pod "b571d1ac-9679-4f13-957e-f87bf892c875" (UID: "b571d1ac-9679-4f13-957e-f87bf892c875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:04.033153 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.033132 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume" pod "fbdb9cc2-807a-437e-96c7-454f224e067e" (UID: "fbdb9cc2-807a-437e-96c7-454f224e067e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:04.033305 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.033290 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b571d1ac-9679-4f13-957e-f87bf892c875-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.033347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.033308 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdb9cc2-807a-437e-96c7-454f224e067e-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.035098 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.035072 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l" pod "b571d1ac-9679-4f13-957e-f87bf892c875" (UID: "b571d1ac-9679-4f13-957e-f87bf892c875"). InnerVolumeSpecName "kube-api-access-z6b5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:04.035233 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.035203 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2" pod "fbdb9cc2-807a-437e-96c7-454f224e067e" (UID: "fbdb9cc2-807a-437e-96c7-454f224e067e"). InnerVolumeSpecName "kube-api-access-42fz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:04.134860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.134822 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b5l\" (UniqueName: \"kubernetes.io/projected/b571d1ac-9679-4f13-957e-f87bf892c875-kube-api-access-z6b5l\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.134860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.134853 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fz2\" (UniqueName: \"kubernetes.io/projected/fbdb9cc2-807a-437e-96c7-454f224e067e-kube-api-access-42fz2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.474063 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.474021 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt"] Oct 30 14:13:04.503090 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.503058 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl"] Oct 30 14:13:04.511904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.511872 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 14:13:04.547941 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.547912 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 14:13:04.639387 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.639352 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:04.640240 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.640214 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8\" (UniqueName: \"kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8\") pod \"4e006fa8-251a-4150-bad3-c7f47233d029\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " Oct 30 14:13:04.640366 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.640348 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume\") pod \"4e006fa8-251a-4150-bad3-c7f47233d029\" (UID: \"4e006fa8-251a-4150-bad3-c7f47233d029\") " Oct 30 14:13:04.640418 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.640395 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume\") pod \"7022b194-fe9d-4290-832b-4380c59eec72\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " Oct 30 14:13:04.640629 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.640588 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh\" (UniqueName: \"kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh\") pod \"7022b194-fe9d-4290-832b-4380c59eec72\" (UID: \"7022b194-fe9d-4290-832b-4380c59eec72\") " Oct 30 14:13:04.640799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.640774 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume" pod "7022b194-fe9d-4290-832b-4380c59eec72" (UID: "7022b194-fe9d-4290-832b-4380c59eec72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:04.640908 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.640830 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume" pod "4e006fa8-251a-4150-bad3-c7f47233d029" (UID: "4e006fa8-251a-4150-bad3-c7f47233d029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:04.641130 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.641107 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e006fa8-251a-4150-bad3-c7f47233d029-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.641175 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.641137 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7022b194-fe9d-4290-832b-4380c59eec72-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.642686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.642655 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8" pod "4e006fa8-251a-4150-bad3-c7f47233d029" (UID: "4e006fa8-251a-4150-bad3-c7f47233d029"). InnerVolumeSpecName "kube-api-access-d9gn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:04.642775 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.642693 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh" pod "7022b194-fe9d-4290-832b-4380c59eec72" (UID: "7022b194-fe9d-4290-832b-4380c59eec72"). InnerVolumeSpecName "kube-api-access-h25vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:04.742065 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.742026 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h25vh\" (UniqueName: \"kubernetes.io/projected/7022b194-fe9d-4290-832b-4380c59eec72-kube-api-access-h25vh\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.742065 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.742056 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9gn8\" (UniqueName: \"kubernetes.io/projected/4e006fa8-251a-4150-bad3-c7f47233d029-kube-api-access-d9gn8\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:04.764074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.764029 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb"] Oct 30 14:13:04.764275 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.764253 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" podUID="15a0869e-b1e7-44a9-bde1-507f205eb920" containerName="maas-api" containerID="cri-o://2c0d50c44a5714850907fe82fd1d5cf00ae2c4a18f16b52669fdac22cef5f900" gracePeriod=30 Oct 30 14:13:04.788107 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.788077 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:04.908035 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.908001 5693 generic.go:356] "Generic (PLEG): container finished" podID="4e006fa8-251a-4150-bad3-c7f47233d029" containerID="3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8" exitCode=0 Oct 30 14:13:04.908237 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.908046 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" Oct 30 14:13:04.908237 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.908081 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" event={"ID":"4e006fa8-251a-4150-bad3-c7f47233d029","Type":"ContainerDied","Data":"3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8"} Oct 30 14:13:04.908237 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.908126 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct" event={"ID":"4e006fa8-251a-4150-bad3-c7f47233d029","Type":"ContainerDied","Data":"e649d9dbc368be7a5d6d910d72db07dd2346cd701f4d8052ec43d331350dfefa"} Oct 30 14:13:04.908237 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.908156 5693 scope.go:122] "RemoveContainer" containerID="3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8" Oct 30 14:13:04.909860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.909742 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" event={"ID":"fbdb9cc2-807a-437e-96c7-454f224e067e","Type":"ContainerDied","Data":"0f23fdda353792ec043c4176409fdd375f3024b9d5e099b02122f9d44ff26ec3"} Oct 30 14:13:04.909860 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.909855 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f" Oct 30 14:13:04.917089 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.917062 5693 generic.go:356] "Generic (PLEG): container finished" podID="15a0869e-b1e7-44a9-bde1-507f205eb920" containerID="2c0d50c44a5714850907fe82fd1d5cf00ae2c4a18f16b52669fdac22cef5f900" exitCode=0 Oct 30 14:13:04.917231 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.917162 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" event={"ID":"15a0869e-b1e7-44a9-bde1-507f205eb920","Type":"ContainerDied","Data":"2c0d50c44a5714850907fe82fd1d5cf00ae2c4a18f16b52669fdac22cef5f900"} Oct 30 14:13:04.918783 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.918738 5693 generic.go:356] "Generic (PLEG): container finished" podID="7022b194-fe9d-4290-832b-4380c59eec72" containerID="81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf" exitCode=0 Oct 30 14:13:04.918891 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.918797 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" Oct 30 14:13:04.918891 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.918845 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" event={"ID":"7022b194-fe9d-4290-832b-4380c59eec72","Type":"ContainerDied","Data":"81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf"} Oct 30 14:13:04.918891 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.918885 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw" event={"ID":"7022b194-fe9d-4290-832b-4380c59eec72","Type":"ContainerDied","Data":"8dd5b79a738bac18ea024ddf6556fcd517c32270fea26d297469c63a124ce8db"} Oct 30 14:13:04.919071 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.918983 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww" Oct 30 14:13:04.921414 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.921396 5693 scope.go:122] "RemoveContainer" containerID="3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8" Oct 30 14:13:04.921744 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:04.921723 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8\": container with ID starting with 3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8 not found: ID does not exist" containerID="3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8" Oct 30 14:13:04.921835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.921749 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8"} err="failed to get container status \"3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8\": rpc error: code = NotFound desc = could not find container \"3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8\": container with ID starting with 3919c63381da6ac1655fff28b0270e72a3a5bd06fb5574fe87e1e892455ae9c8 not found: ID does not exist" Oct 30 14:13:04.921835 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.921764 5693 scope.go:122] "RemoveContainer" containerID="744ca13c4555f220af2005cca159f4e3a582a45a19392f48734e4c4fb6f07403" Oct 30 14:13:04.929768 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.929742 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:04.931143 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.931128 5693 scope.go:122] "RemoveContainer" containerID="81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf" Oct 30 14:13:04.941189 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.941162 5693 scope.go:122] "RemoveContainer" containerID="81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf" Oct 30 14:13:04.941536 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:04.941507 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf\": container with ID starting with 81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf not found: ID does not exist" containerID="81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf" Oct 30 14:13:04.941740 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.941546 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf"} err="failed to get container status \"81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf\": rpc error: code = NotFound desc = could not find container \"81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf\": container with ID starting with 81a0795ead6f69839eb570e2408ceef8f0009b9d72b69f3fdae407771344ddbf not found: ID does not exist" Oct 30 14:13:04.957022 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.956993 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct"] Oct 30 14:13:04.962059 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.962030 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c684d2-57b55dd47d-jntct"] Oct 30 14:13:04.994718 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.994686 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f"] Oct 30 14:13:04.998924 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:04.998895 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52af49-798d79d7dc-d2r7f"] Oct 30 14:13:05.027162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.027132 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 14:13:05.048829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.048793 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww"] Oct 30 14:13:05.051479 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.051428 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52af49-7db48668c8-xpvww"] Oct 30 14:13:05.087896 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.087859 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw"] Oct 30 14:13:05.092200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.092171 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c684d2-76d99589ff-95jxw"] Oct 30 14:13:05.147502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.145910 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca\") pod \"15a0869e-b1e7-44a9-bde1-507f205eb920\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " Oct 30 14:13:05.147502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.145986 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2\" (UniqueName: \"kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2\") pod \"15a0869e-b1e7-44a9-bde1-507f205eb920\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " Oct 30 14:13:05.147502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.146103 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls\") pod \"15a0869e-b1e7-44a9-bde1-507f205eb920\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " Oct 30 14:13:05.147502 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.146191 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs\") pod \"15a0869e-b1e7-44a9-bde1-507f205eb920\" (UID: \"15a0869e-b1e7-44a9-bde1-507f205eb920\") " Oct 30 14:13:05.149433 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.149392 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs" pod "15a0869e-b1e7-44a9-bde1-507f205eb920" (UID: "15a0869e-b1e7-44a9-bde1-507f205eb920"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:05.149658 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.149459 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca" pod "15a0869e-b1e7-44a9-bde1-507f205eb920" (UID: "15a0869e-b1e7-44a9-bde1-507f205eb920"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:05.152177 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.152139 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2" pod "15a0869e-b1e7-44a9-bde1-507f205eb920" (UID: "15a0869e-b1e7-44a9-bde1-507f205eb920"). InnerVolumeSpecName "kube-api-access-qbhs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:05.152306 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.152233 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls" pod "15a0869e-b1e7-44a9-bde1-507f205eb920" (UID: "15a0869e-b1e7-44a9-bde1-507f205eb920"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:05.247545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.247514 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-maas-api-tls\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:05.247545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.247541 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a0869e-b1e7-44a9-bde1-507f205eb920-metrics-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:05.247545 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.247551 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0869e-b1e7-44a9-bde1-507f205eb920-openshift-service-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:05.247864 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.247560 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhs2\" (UniqueName: \"kubernetes.io/projected/15a0869e-b1e7-44a9-bde1-507f205eb920-kube-api-access-qbhs2\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:05.524181 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.524146 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:05.928611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.928521 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" event={"ID":"15a0869e-b1e7-44a9-bde1-507f205eb920","Type":"ContainerDied","Data":"aed02a48cdaef8f38deb590d3ce2f12706bae22958b6e1cc3204b09429e9076d"} Oct 30 14:13:05.928611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.928557 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb" Oct 30 14:13:05.928611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.928580 5693 scope.go:122] "RemoveContainer" containerID="2c0d50c44a5714850907fe82fd1d5cf00ae2c4a18f16b52669fdac22cef5f900" Oct 30 14:13:05.987437 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:05.987394 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb"] Oct 30 14:13:06.002099 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:06.002066 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22af91-cc4f9bc48-bvzwb"] Oct 30 14:13:06.252985 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:06.252953 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a0869e-b1e7-44a9-bde1-507f205eb920" path="/var/lib/kubelet/pods/15a0869e-b1e7-44a9-bde1-507f205eb920/volumes" Oct 30 14:13:06.253392 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:06.253377 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" path="/var/lib/kubelet/pods/4e006fa8-251a-4150-bad3-c7f47233d029/volumes" Oct 30 14:13:06.253795 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:06.253778 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7022b194-fe9d-4290-832b-4380c59eec72" path="/var/lib/kubelet/pods/7022b194-fe9d-4290-832b-4380c59eec72/volumes" Oct 30 14:13:06.254423 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:06.254408 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b571d1ac-9679-4f13-957e-f87bf892c875" path="/var/lib/kubelet/pods/b571d1ac-9679-4f13-957e-f87bf892c875/volumes" Oct 30 14:13:06.254800 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:06.254786 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdb9cc2-807a-437e-96c7-454f224e067e" path="/var/lib/kubelet/pods/fbdb9cc2-807a-437e-96c7-454f224e067e/volumes" Oct 30 14:13:07.215136 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:07.215092 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:08.086290 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.086248 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" containerID="cri-o://61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25" gracePeriod=45 Oct 30 14:13:08.096923 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.096887 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused" Oct 30 14:13:08.336262 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.336236 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 14:13:08.479017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.478990 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58\" (UniqueName: \"kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58\") pod \"e843b93f-d34c-482f-a433-55576dd790c1\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " Oct 30 14:13:08.479200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.479029 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs\") pod \"e843b93f-d34c-482f-a433-55576dd790c1\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " Oct 30 14:13:08.479200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.479053 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home\") pod \"e843b93f-d34c-482f-a433-55576dd790c1\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " Oct 30 14:13:08.479200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.479161 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir\") pod \"e843b93f-d34c-482f-a433-55576dd790c1\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " Oct 30 14:13:08.479200 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.479190 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache\") pod \"e843b93f-d34c-482f-a433-55576dd790c1\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " Oct 30 14:13:08.479469 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.479330 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm\") pod \"e843b93f-d34c-482f-a433-55576dd790c1\" (UID: \"e843b93f-d34c-482f-a433-55576dd790c1\") " Oct 30 14:13:08.479469 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.479374 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home" pod "e843b93f-d34c-482f-a433-55576dd790c1" (UID: "e843b93f-d34c-482f-a433-55576dd790c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:08.480092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.480057 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir" pod "e843b93f-d34c-482f-a433-55576dd790c1" (UID: "e843b93f-d34c-482f-a433-55576dd790c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:08.480092 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.480057 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache" pod "e843b93f-d34c-482f-a433-55576dd790c1" (UID: "e843b93f-d34c-482f-a433-55576dd790c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:08.480664 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.480640 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:08.480748 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.480670 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:08.480748 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.480696 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:08.493510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.493480 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs" pod "e843b93f-d34c-482f-a433-55576dd790c1" (UID: "e843b93f-d34c-482f-a433-55576dd790c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:08.493510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.493495 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm" pod "e843b93f-d34c-482f-a433-55576dd790c1" (UID: "e843b93f-d34c-482f-a433-55576dd790c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:08.493693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.493502 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58" pod "e843b93f-d34c-482f-a433-55576dd790c1" (UID: "e843b93f-d34c-482f-a433-55576dd790c1"). InnerVolumeSpecName "kube-api-access-b9t58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:08.581723 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.581685 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e843b93f-d34c-482f-a433-55576dd790c1-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:08.581723 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.581717 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9t58\" (UniqueName: \"kubernetes.io/projected/e843b93f-d34c-482f-a433-55576dd790c1-kube-api-access-b9t58\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:08.581723 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.581729 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e843b93f-d34c-482f-a433-55576dd790c1-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:08.948229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.948140 5693 generic.go:356] "Generic (PLEG): container finished" podID="e843b93f-d34c-482f-a433-55576dd790c1" containerID="61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25" exitCode=0 Oct 30 14:13:08.948229 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.948199 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" Oct 30 14:13:08.948501 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.948222 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" event={"ID":"e843b93f-d34c-482f-a433-55576dd790c1","Type":"ContainerDied","Data":"61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25"} Oct 30 14:13:08.948501 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.948265 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p" event={"ID":"e843b93f-d34c-482f-a433-55576dd790c1","Type":"ContainerDied","Data":"888f2c3161b2dada640f949b9687114706c3b4844cfec205b5a6cd073cbfc83e"} Oct 30 14:13:08.948501 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.948283 5693 scope.go:122] "RemoveContainer" containerID="61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25" Oct 30 14:13:08.957677 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.957656 5693 scope.go:122] "RemoveContainer" containerID="61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25" Oct 30 14:13:08.957917 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:08.957899 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25\": container with ID starting with 61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25 not found: ID does not exist" containerID="61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25" Oct 30 14:13:08.957990 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:08.957925 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25"} err="failed to get container status \"61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25\": rpc error: code = NotFound desc = could not find container \"61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25\": container with ID starting with 61c53f86a013c4448d8340acad96765bab6a22e65b22016d5616f1de3fdfbd25 not found: ID does not exist" Oct 30 14:13:09.000670 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.000635 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p"] Oct 30 14:13:09.002350 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.002318 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/premium-w1-52af49-kserve-5b9df45d4d-lrp4p"] Oct 30 14:13:09.118349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.118320 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:09.269675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.269640 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:09.475436 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.475394 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerName="payload-processing" containerID="cri-o://107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6" gracePeriod=25 Oct 30 14:13:09.504110 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.504067 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerName="payload-pre-processing" containerID="cri-o://62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8" gracePeriod=25 Oct 30 14:13:09.746558 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:09.746513 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:10.253892 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:10.253848 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e843b93f-d34c-482f-a433-55576dd790c1" path="/var/lib/kubelet/pods/e843b93f-d34c-482f-a433-55576dd790c1/volumes" Oct 30 14:13:11.007056 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:11.007022 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:12.764957 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:12.764922 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:14.514488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:14.514455 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:15.357402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:15.357359 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:13:15.357610 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:15.357402 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:13:16.201909 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:16.201867 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:16.872253 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:16.872217 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:17.033661 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:17.033624 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:17.198234 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:17.198144 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp"] Oct 30 14:13:17.198407 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:17.198371 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" containerID="cri-o://8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793" gracePeriod=30 Oct 30 14:13:17.278234 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:17.278168 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:17.729041 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:17.728996 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:17.881850 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:17.881815 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:18.042347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:18.042270 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:18.202096 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:18.202062 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:19.476983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:19.476943 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:19.874719 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:19.874634 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:20.053019 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:20.052980 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:20.200265 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:20.200183 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:20.466524 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:20.466438 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:21.713643 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:21.713592 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:22.443313 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.443286 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 14:13:22.512309 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512278 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512317 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512347 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512401 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512452 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512507 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512491 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512858 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512531 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512858 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512632 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:22.512858 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512658 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.512858 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512670 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:22.513170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512920 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:22.513170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.512953 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:22.513170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.513022 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-data\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.513170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.513041 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.513170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.513056 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-workload-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.513170 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.513095 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:22.514743 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.514710 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:22.514874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.514759 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "kube-api-access-pnnxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:22.514874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.514815 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:22.613807 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.613772 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo\") pod \"cf039ca7-5fc7-4310-8c29-685bc70445d1\" (UID: \"cf039ca7-5fc7-4310-8c29-685bc70445d1\") " Oct 30 14:13:22.614162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.614139 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-token\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.614162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.614162 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-credential-socket\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.614291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.614172 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnnxl\" (UniqueName: \"kubernetes.io/projected/cf039ca7-5fc7-4310-8c29-685bc70445d1-kube-api-access-pnnxl\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.614291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.614181 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-envoy\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.614291 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.614191 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf039ca7-5fc7-4310-8c29-685bc70445d1-istiod-ca-cert\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.615878 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.615854 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo" pod "cf039ca7-5fc7-4310-8c29-685bc70445d1" (UID: "cf039ca7-5fc7-4310-8c29-685bc70445d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:13:22.654585 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.654550 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk"] Oct 30 14:13:22.677488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.677457 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98"] Oct 30 14:13:22.715472 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.715436 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf039ca7-5fc7-4310-8c29-685bc70445d1-istio-podinfo\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.867718 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:22.867635 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:23.024159 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.024123 5693 generic.go:356] "Generic (PLEG): container finished" podID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerID="8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793" exitCode=0 Oct 30 14:13:23.024338 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.024171 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" Oct 30 14:13:23.024338 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.024207 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" event={"ID":"cf039ca7-5fc7-4310-8c29-685bc70445d1","Type":"ContainerDied","Data":"8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793"} Oct 30 14:13:23.024338 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.024262 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp" event={"ID":"cf039ca7-5fc7-4310-8c29-685bc70445d1","Type":"ContainerDied","Data":"22410d6aab011d3a761413b421f44ccfdf1b1ca01c4c4d2ec3c1ff7ded646af3"} Oct 30 14:13:23.024338 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.024294 5693 scope.go:122] "RemoveContainer" containerID="8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793" Oct 30 14:13:23.035153 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.035134 5693 scope.go:122] "RemoveContainer" containerID="30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4" Oct 30 14:13:23.043778 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.043755 5693 scope.go:122] "RemoveContainer" containerID="8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793" Oct 30 14:13:23.044113 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:23.044089 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793\": container with ID starting with 8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793 not found: ID does not exist" containerID="8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793" Oct 30 14:13:23.044174 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.044123 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793"} err="failed to get container status \"8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793\": rpc error: code = NotFound desc = could not find container \"8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793\": container with ID starting with 8ecc6ceb3472aef2eca587713b3efeae74a9527d51ec7aad30d3f9ea301ff793 not found: ID does not exist" Oct 30 14:13:23.044174 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.044141 5693 scope.go:122] "RemoveContainer" containerID="30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4" Oct 30 14:13:23.044449 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:23.044424 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4\": container with ID starting with 30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4 not found: ID does not exist" containerID="30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4" Oct 30 14:13:23.044508 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.044447 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4"} err="failed to get container status \"30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4\": rpc error: code = NotFound desc = could not find container \"30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4\": container with ID starting with 30dd922f0c479c8a48ad6138463679d3cea1f5b98f1707585bfd40817859bdc4 not found: ID does not exist" Oct 30 14:13:23.056212 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:23.056184 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:13:23.073490 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.073432 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp"] Oct 30 14:13:23.075549 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.075526 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-rldmp"] Oct 30 14:13:23.415670 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.415634 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:23.866427 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:23.866382 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:24.187359 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:24.187269 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:24.254102 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:24.254071 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" path="/var/lib/kubelet/pods/cf039ca7-5fc7-4310-8c29-685bc70445d1/volumes" Oct 30 14:13:24.341298 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:24.341269 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:24.527360 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:24.527324 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:25.093686 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:25.093653 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:25.293727 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:25.293692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:25.357561 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:25.357468 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:13:25.357748 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:25.357474 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:13:26.726587 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:26.726546 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:26.891448 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:26.891409 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:27.044513 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:27.044424 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:28.302045 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:28.301996 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b"] Oct 30 14:13:28.371646 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:28.370046 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:28.531454 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:28.531401 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:29.756351 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.756320 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 14:13:29.767406 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767376 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c"] Oct 30 14:13:29.767816 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767797 5693 cpu_manager.go:414] "containerMap: removing container" podUID="98d025ec-a9d7-4b3b-884d-80e52a801a92" containerName="maas-api" Oct 30 14:13:29.767816 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767813 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d025ec-a9d7-4b3b-884d-80e52a801a92" containerName="maas-api" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767827 5693 cpu_manager.go:414] "containerMap: removing container" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767833 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767842 5693 cpu_manager.go:414] "containerMap: removing container" podUID="201a039d-07be-4f7a-b194-7a536c6fdfcc" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767847 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201a039d-07be-4f7a-b194-7a536c6fdfcc" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767855 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767860 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767869 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767874 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767880 5693 cpu_manager.go:414] "containerMap: removing container" podUID="70c6f506-deae-4d9e-85c7-f4507adba4b2" containerName="maas-api" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767884 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c6f506-deae-4d9e-85c7-f4507adba4b2" containerName="maas-api" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767892 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7014e751-7cca-4fe1-bfc4-99170da7256a" containerName="main" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767897 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7014e751-7cca-4fe1-bfc4-99170da7256a" containerName="main" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767903 5693 cpu_manager.go:414] "containerMap: removing container" podUID="15a0869e-b1e7-44a9-bde1-507f205eb920" containerName="maas-api" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767908 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a0869e-b1e7-44a9-bde1-507f205eb920" containerName="maas-api" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767914 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767919 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767927 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767932 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767939 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767944 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767951 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerName="payload-pre-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767958 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerName="payload-pre-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767969 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767975 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767983 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3789b872-e323-4b81-a952-6783a7209ece" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767991 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3789b872-e323-4b81-a952-6783a7209ece" containerName="payload-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.767999 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768007 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768013 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768018 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768027 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fbdb9cc2-807a-437e-96c7-454f224e067e" containerName="payload-pre-processing" Oct 30 14:13:29.768017 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768033 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdb9cc2-807a-437e-96c7-454f224e067e" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768043 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768051 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768062 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7022b194-fe9d-4290-832b-4380c59eec72" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768066 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7022b194-fe9d-4290-832b-4380c59eec72" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768074 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768078 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768085 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768089 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768099 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1a3e16f5-7af6-428a-a80e-fc16c17b6f51" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768104 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3e16f5-7af6-428a-a80e-fc16c17b6f51" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768110 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768116 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768125 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768131 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768138 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2a1169eb-6098-414b-a78e-4b7320f35189" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768144 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1169eb-6098-414b-a78e-4b7320f35189" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768154 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768160 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768169 5693 cpu_manager.go:414] "containerMap: removing container" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768174 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768182 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b571d1ac-9679-4f13-957e-f87bf892c875" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768186 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b571d1ac-9679-4f13-957e-f87bf892c875" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768193 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768200 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768211 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768218 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768225 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7878c52d-f84f-4d12-8bdd-937eb5a9644a" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768232 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7878c52d-f84f-4d12-8bdd-937eb5a9644a" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768243 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7b98b358-3b68-43b6-93db-d585fb5a2d82" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768249 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b98b358-3b68-43b6-93db-d585fb5a2d82" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768263 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768269 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768280 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768289 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768305 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a41b1373-0002-4a40-9a9d-abff679404c0" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768312 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41b1373-0002-4a40-9a9d-abff679404c0" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768328 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768335 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768348 5693 cpu_manager.go:414] "containerMap: removing container" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768355 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768368 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768375 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768384 5693 cpu_manager.go:414] "containerMap: removing container" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768390 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768403 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768410 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768426 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d1b5a289-4aea-4a69-9544-a3326448333a" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768433 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b5a289-4aea-4a69-9544-a3326448333a" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768444 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768452 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768464 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768470 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768481 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768488 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768587 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d967b1c-4a62-49ff-bbc0-0e5900fd7d14" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768628 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d975ade9-c5cf-4335-9d2f-3ead72ae9bb2" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768639 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41b1373-0002-4a40-9a9d-abff679404c0" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768646 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768652 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768659 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f8754c-1a5a-4ea0-86eb-ad4c31e876f5" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768666 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4529cf2d-afc8-4338-b21e-65bdf6cdf76e" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768671 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="895e8e9a-af7b-4d49-9132-5c494886f00f" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768677 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b25af17-7179-427d-b5f8-df3fd9f434b7" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768683 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7772c4-fda4-4bf7-9cd1-a419d07aff95" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768690 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b5a289-4aea-4a69-9544-a3326448333a" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768696 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0fbda9-c630-43fa-ab55-6d6275a24187" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768704 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a394b55a-96df-4246-86b1-bd9d3f98eb18" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768711 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a41edd2-d1ed-4bc9-95bb-8491eab9dd12" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768720 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d025ec-a9d7-4b3b-884d-80e52a801a92" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768726 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="201a039d-07be-4f7a-b194-7a536c6fdfcc" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768733 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768739 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768745 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c6f506-deae-4d9e-85c7-f4507adba4b2" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768750 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7014e751-7cca-4fe1-bfc4-99170da7256a" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768755 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a0869e-b1e7-44a9-bde1-507f205eb920" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768761 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768766 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5156f402-98f2-4349-9956-4874b8b04ed4" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768772 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768777 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffcbe21-629f-402e-9847-02108c1a0392" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768784 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29f0d1c-172c-4b0b-9de5-29bcc6a93818" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768792 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3789b872-e323-4b81-a952-6783a7209ece" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768799 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cffe0f8-7282-4e8f-ae05-d15c86ef35d4" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768804 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdb9cc2-807a-437e-96c7-454f224e067e" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768809 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9623ae-3597-4add-97e7-a21f3b38c6b6" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768815 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7022b194-fe9d-4290-832b-4380c59eec72" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768824 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fedb5b6c-e1f5-4fd6-a8e3-dcdb40aa42bb" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768831 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3e16f5-7af6-428a-a80e-fc16c17b6f51" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768836 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e843b93f-d34c-482f-a433-55576dd790c1" containerName="main" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768841 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768846 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1169eb-6098-414b-a78e-4b7320f35189" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768851 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768858 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="be62a5bc-0e09-4d0e-a514-14b28e9b98df" containerName="payload-pre-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768868 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a928a3-4554-425d-a2e5-38a118e6729e" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768874 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768879 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b571d1ac-9679-4f13-957e-f87bf892c875" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768884 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e006fa8-251a-4150-bad3-c7f47233d029" containerName="payload-processing" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768889 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7878c52d-f84f-4d12-8bdd-937eb5a9644a" containerName="maas-api" Oct 30 14:13:29.770105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.768894 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b98b358-3b68-43b6-93db-d585fb5a2d82" containerName="maas-api" Oct 30 14:13:29.775327 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.772149 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.781147 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.781126 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 14:13:29.786540 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.786519 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:13:29.786664 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.786624 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\"" Oct 30 14:13:29.789769 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.789745 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume\") pod \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " Oct 30 14:13:29.790283 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.789932 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf\" (UniqueName: \"kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf\") pod \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\" (UID: \"6efcd45d-f7fe-49dc-b448-5d89f945a76c\") " Oct 30 14:13:29.790819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.790796 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume" pod "6efcd45d-f7fe-49dc-b448-5d89f945a76c" (UID: "6efcd45d-f7fe-49dc-b448-5d89f945a76c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:29.792451 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.792428 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf" pod "6efcd45d-f7fe-49dc-b448-5d89f945a76c" (UID: "6efcd45d-f7fe-49dc-b448-5d89f945a76c"). InnerVolumeSpecName "kube-api-access-szstf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:29.795125 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.795103 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c"] Oct 30 14:13:29.891258 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891227 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume\") pod \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " Oct 30 14:13:29.891426 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891292 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l\" (UniqueName: \"kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l\") pod \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\" (UID: \"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65\") " Oct 30 14:13:29.891514 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891500 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.891562 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891547 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzl42\" (UniqueName: \"kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.891629 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891578 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.891699 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891666 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume" pod "56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" (UID: "56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:29.891829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891768 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efcd45d-f7fe-49dc-b448-5d89f945a76c-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:29.891829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891782 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szstf\" (UniqueName: \"kubernetes.io/projected/6efcd45d-f7fe-49dc-b448-5d89f945a76c-kube-api-access-szstf\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:29.891829 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.891791 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:29.893382 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.893357 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l" pod "56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" (UID: "56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65"). InnerVolumeSpecName "kube-api-access-xv88l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:29.992849 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.992775 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.992995 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.992857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzl42\" (UniqueName: \"kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.992995 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.992893 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.993082 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.993042 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv88l\" (UniqueName: \"kubernetes.io/projected/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65-kube-api-access-xv88l\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:29.993474 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.993452 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:29.995288 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:29.995262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:30.012353 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.012328 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzl42\" (UniqueName: \"kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42\") pod \"maas-api-revoke-sub-13946755-zsf8c\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:30.063822 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.063789 5693 generic.go:356] "Generic (PLEG): container finished" podID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerID="62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8" exitCode=0 Oct 30 14:13:30.063973 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.063840 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" Oct 30 14:13:30.063973 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.063876 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" event={"ID":"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65","Type":"ContainerDied","Data":"62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8"} Oct 30 14:13:30.063973 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.063932 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl" event={"ID":"56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65","Type":"ContainerDied","Data":"2574fda2431d3cf58e0a8a6d1a41b511b5dcc430f3a5afc749f2c00c3caa1ffd"} Oct 30 14:13:30.063973 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.063961 5693 scope.go:122] "RemoveContainer" containerID="62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8" Oct 30 14:13:30.065370 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.065347 5693 generic.go:356] "Generic (PLEG): container finished" podID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" containerID="107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6" exitCode=0 Oct 30 14:13:30.065504 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.065381 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" event={"ID":"6efcd45d-f7fe-49dc-b448-5d89f945a76c","Type":"ContainerDied","Data":"107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6"} Oct 30 14:13:30.065504 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.065424 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" event={"ID":"6efcd45d-f7fe-49dc-b448-5d89f945a76c","Type":"ContainerDied","Data":"d119f49a8c41743084a573d953a2e665101891dc5ca19823caae2771c88e9bd4"} Oct 30 14:13:30.065504 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.065451 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt" Oct 30 14:13:30.073157 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.073126 5693 scope.go:122] "RemoveContainer" containerID="62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8" Oct 30 14:13:30.073452 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:30.073433 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8\": container with ID starting with 62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8 not found: ID does not exist" containerID="62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8" Oct 30 14:13:30.073508 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.073462 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8"} err="failed to get container status \"62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8\": rpc error: code = NotFound desc = could not find container \"62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8\": container with ID starting with 62dacad9c1d52d2f2b25e1020d34981378061bcb28a3b02c89f08c2d061fa9b8 not found: ID does not exist" Oct 30 14:13:30.073508 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.073480 5693 scope.go:122] "RemoveContainer" containerID="107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6" Oct 30 14:13:30.082370 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.082354 5693 scope.go:122] "RemoveContainer" containerID="107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6" Oct 30 14:13:30.082633 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:30.082611 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6\": container with ID starting with 107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6 not found: ID does not exist" containerID="107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6" Oct 30 14:13:30.082693 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.082645 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6"} err="failed to get container status \"107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6\": rpc error: code = NotFound desc = could not find container \"107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6\": container with ID starting with 107cb620ada867f7719fecdbd599b5fa9463677427c2d9b22ced8eab01f561a6 not found: ID does not exist" Oct 30 14:13:30.089901 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.089883 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:30.114748 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.114716 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt"] Oct 30 14:13:30.119132 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.119103 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-22af91-6ff8c8f956-pzsqt"] Oct 30 14:13:30.153224 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.153190 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl"] Oct 30 14:13:30.156035 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.155821 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22af91-9b87df7d7-rrggl"] Oct 30 14:13:30.253119 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.253043 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" path="/var/lib/kubelet/pods/56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65/volumes" Oct 30 14:13:30.253480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.253464 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efcd45d-f7fe-49dc-b448-5d89f945a76c" path="/var/lib/kubelet/pods/6efcd45d-f7fe-49dc-b448-5d89f945a76c/volumes" Oct 30 14:13:30.386864 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:30.386823 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c"] Oct 30 14:13:30.391060 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:13:30.391032 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddd73e0_4506_472b_aea2_be55a188a936.slice/crio-c3537ee65a93c36e68f544f5d7d0b12753055dd2bd009baa142c6f95da60188a WatchSource:0}: Error finding container c3537ee65a93c36e68f544f5d7d0b12753055dd2bd009baa142c6f95da60188a: Status 404 returned error can't find the container with id c3537ee65a93c36e68f544f5d7d0b12753055dd2bd009baa142c6f95da60188a Oct 30 14:13:31.072013 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:31.071970 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" event={"ID":"eddd73e0-4506-472b-aea2-be55a188a936","Type":"ContainerStarted","Data":"4aec6ba858e2c93c6ad46881dec4c7d5ea5b7388373144a637ff757a560b13a0"} Oct 30 14:13:31.072771 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:31.072038 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" event={"ID":"eddd73e0-4506-472b-aea2-be55a188a936","Type":"ContainerStarted","Data":"c3537ee65a93c36e68f544f5d7d0b12753055dd2bd009baa142c6f95da60188a"} Oct 30 14:13:31.109218 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:31.109172 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" podStartSLOduration=2.109159973 podStartE2EDuration="2.109159973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:31.10854791 +0000 UTC m=+4186.975166087" watchObservedRunningTime="2026-10-30 14:13:31.109159973 +0000 UTC m=+4186.975778145" Oct 30 14:13:31.986222 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:31.986188 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:32.081013 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:32.080970 5693 generic.go:356] "Generic (PLEG): container finished" podID="eddd73e0-4506-472b-aea2-be55a188a936" containerID="4aec6ba858e2c93c6ad46881dec4c7d5ea5b7388373144a637ff757a560b13a0" exitCode=0 Oct 30 14:13:32.081402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:32.081051 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" event={"ID":"eddd73e0-4506-472b-aea2-be55a188a936","Type":"ContainerDied","Data":"4aec6ba858e2c93c6ad46881dec4c7d5ea5b7388373144a637ff757a560b13a0"} Oct 30 14:13:33.224450 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.224427 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:33.303615 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.303568 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" containerName="payload-processing" containerID="cri-o://71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1" gracePeriod=25 Oct 30 14:13:33.327637 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.327587 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca\") pod \"eddd73e0-4506-472b-aea2-be55a188a936\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " Oct 30 14:13:33.327777 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.327745 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42\" (UniqueName: \"kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42\") pod \"eddd73e0-4506-472b-aea2-be55a188a936\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " Oct 30 14:13:33.327819 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.327799 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca\") pod \"eddd73e0-4506-472b-aea2-be55a188a936\" (UID: \"eddd73e0-4506-472b-aea2-be55a188a936\") " Oct 30 14:13:33.327894 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.327868 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca" pod "eddd73e0-4506-472b-aea2-be55a188a936" (UID: "eddd73e0-4506-472b-aea2-be55a188a936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:33.328162 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.328144 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-openshift-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:33.329675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.329652 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42" pod "eddd73e0-4506-472b-aea2-be55a188a936" (UID: "eddd73e0-4506-472b-aea2-be55a188a936"). InnerVolumeSpecName "kube-api-access-hzl42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:33.329734 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.329710 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca" pod "eddd73e0-4506-472b-aea2-be55a188a936" (UID: "eddd73e0-4506-472b-aea2-be55a188a936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:33.429383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.429320 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eddd73e0-4506-472b-aea2-be55a188a936-maas-api-xks-ca\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:33.429383 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:33.429345 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzl42\" (UniqueName: \"kubernetes.io/projected/eddd73e0-4506-472b-aea2-be55a188a936-kube-api-access-hzl42\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:34.092577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:34.092546 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" Oct 30 14:13:34.092782 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:34.092581 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13946755-zsf8c" event={"ID":"eddd73e0-4506-472b-aea2-be55a188a936","Type":"ContainerDied","Data":"c3537ee65a93c36e68f544f5d7d0b12753055dd2bd009baa142c6f95da60188a"} Oct 30 14:13:34.092782 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:34.092648 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3537ee65a93c36e68f544f5d7d0b12753055dd2bd009baa142c6f95da60188a" Oct 30 14:13:37.117002 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:37.116963 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:13:37.674594 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:37.674547 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" containerID="cri-o://9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da" gracePeriod=45 Oct 30 14:13:37.697130 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:37.697076 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" containerID="cri-o://ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9" gracePeriod=45 Oct 30 14:13:37.940577 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:37.940517 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 14:13:37.965231 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:37.965193 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 14:13:38.077588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077551 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh\" (UniqueName: \"kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh\") pod \"1990209a-3cb1-4793-9d68-ae869455f487\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " Oct 30 14:13:38.077790 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077618 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs\") pod \"f4b78d06-b249-4247-b0e7-78acefe36d87\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " Oct 30 14:13:38.077790 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077652 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm\") pod \"f4b78d06-b249-4247-b0e7-78acefe36d87\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " Oct 30 14:13:38.077790 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077696 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm\") pod \"1990209a-3cb1-4793-9d68-ae869455f487\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " Oct 30 14:13:38.077790 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077739 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache\") pod \"1990209a-3cb1-4793-9d68-ae869455f487\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " Oct 30 14:13:38.078051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077794 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir\") pod \"f4b78d06-b249-4247-b0e7-78acefe36d87\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " Oct 30 14:13:38.078051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077827 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home\") pod \"1990209a-3cb1-4793-9d68-ae869455f487\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " Oct 30 14:13:38.078051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077860 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir\") pod \"1990209a-3cb1-4793-9d68-ae869455f487\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " Oct 30 14:13:38.078051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.077889 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs\" (UniqueName: \"kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs\") pod \"1990209a-3cb1-4793-9d68-ae869455f487\" (UID: \"1990209a-3cb1-4793-9d68-ae869455f487\") " Oct 30 14:13:38.078051 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078006 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home\") pod \"f4b78d06-b249-4247-b0e7-78acefe36d87\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " Oct 30 14:13:38.078332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache\") pod \"f4b78d06-b249-4247-b0e7-78acefe36d87\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " Oct 30 14:13:38.078332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078106 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache" pod "1990209a-3cb1-4793-9d68-ae869455f487" (UID: "1990209a-3cb1-4793-9d68-ae869455f487"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.078332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078129 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd\" (UniqueName: \"kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd\") pod \"f4b78d06-b249-4247-b0e7-78acefe36d87\" (UID: \"f4b78d06-b249-4247-b0e7-78acefe36d87\") " Oct 30 14:13:38.078332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078133 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home" pod "1990209a-3cb1-4793-9d68-ae869455f487" (UID: "1990209a-3cb1-4793-9d68-ae869455f487"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.078560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078298 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir" pod "1990209a-3cb1-4793-9d68-ae869455f487" (UID: "1990209a-3cb1-4793-9d68-ae869455f487"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.078560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078342 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home" pod "f4b78d06-b249-4247-b0e7-78acefe36d87" (UID: "f4b78d06-b249-4247-b0e7-78acefe36d87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.078560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078357 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir" pod "f4b78d06-b249-4247-b0e7-78acefe36d87" (UID: "f4b78d06-b249-4247-b0e7-78acefe36d87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.078560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078536 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache" pod "f4b78d06-b249-4247-b0e7-78acefe36d87" (UID: "f4b78d06-b249-4247-b0e7-78acefe36d87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.078560 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078556 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.078879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078575 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.078879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078591 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.078879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078618 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-home\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.078879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.078629 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-tmp-dir\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.080409 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.080379 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm" pod "f4b78d06-b249-4247-b0e7-78acefe36d87" (UID: "f4b78d06-b249-4247-b0e7-78acefe36d87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.080409 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.080389 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs" pod "f4b78d06-b249-4247-b0e7-78acefe36d87" (UID: "f4b78d06-b249-4247-b0e7-78acefe36d87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:38.080587 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.080409 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm" pod "1990209a-3cb1-4793-9d68-ae869455f487" (UID: "1990209a-3cb1-4793-9d68-ae869455f487"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:38.080587 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.080579 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs" pod "1990209a-3cb1-4793-9d68-ae869455f487" (UID: "1990209a-3cb1-4793-9d68-ae869455f487"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:38.080707 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.080640 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh" pod "1990209a-3cb1-4793-9d68-ae869455f487" (UID: "1990209a-3cb1-4793-9d68-ae869455f487"). InnerVolumeSpecName "kube-api-access-gwkfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:38.080757 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.080743 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd" pod "f4b78d06-b249-4247-b0e7-78acefe36d87" (UID: "f4b78d06-b249-4247-b0e7-78acefe36d87"). InnerVolumeSpecName "kube-api-access-685gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:38.115194 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.115165 5693 generic.go:356] "Generic (PLEG): container finished" podID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerID="9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da" exitCode=0 Oct 30 14:13:38.115324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.115222 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" Oct 30 14:13:38.115324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.115239 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" event={"ID":"f4b78d06-b249-4247-b0e7-78acefe36d87","Type":"ContainerDied","Data":"9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da"} Oct 30 14:13:38.115324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.115276 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk" event={"ID":"f4b78d06-b249-4247-b0e7-78acefe36d87","Type":"ContainerDied","Data":"58d72d95543e1e974297daad31b57a71af5870c1b685976ecb45ac3e87cb3455"} Oct 30 14:13:38.115324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.115297 5693 scope.go:122] "RemoveContainer" containerID="9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da" Oct 30 14:13:38.116799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.116775 5693 generic.go:356] "Generic (PLEG): container finished" podID="1990209a-3cb1-4793-9d68-ae869455f487" containerID="ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9" exitCode=0 Oct 30 14:13:38.116899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.116817 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" Oct 30 14:13:38.116899 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.116862 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" event={"ID":"1990209a-3cb1-4793-9d68-ae869455f487","Type":"ContainerDied","Data":"ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9"} Oct 30 14:13:38.116996 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.116906 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98" event={"ID":"1990209a-3cb1-4793-9d68-ae869455f487","Type":"ContainerDied","Data":"40bc1a8efc67a5a35b42dbe39181c29832e459594fbdf655a3995b70e5bbdf48"} Oct 30 14:13:38.127688 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.127497 5693 scope.go:122] "RemoveContainer" containerID="9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da" Oct 30 14:13:38.128093 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:38.127838 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da\": container with ID starting with 9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da not found: ID does not exist" containerID="9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da" Oct 30 14:13:38.128093 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.127874 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da"} err="failed to get container status \"9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da\": rpc error: code = NotFound desc = could not find container \"9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da\": container with ID starting with 9a9e6e59e649fa5f2097ae038aa5aa6c11c65e2dc87462da38f41b5f062e19da not found: ID does not exist" Oct 30 14:13:38.128093 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.127902 5693 scope.go:122] "RemoveContainer" containerID="ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9" Oct 30 14:13:38.136555 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.136532 5693 scope.go:122] "RemoveContainer" containerID="ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9" Oct 30 14:13:38.136882 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:38.136862 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9\": container with ID starting with ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9 not found: ID does not exist" containerID="ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9" Oct 30 14:13:38.136970 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.136892 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9"} err="failed to get container status \"ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9\": rpc error: code = NotFound desc = could not find container \"ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9\": container with ID starting with ad707e5939cb1bea4bcf94c0ad5d5cb452ac49d39f576ff866609e3bbfa635b9 not found: ID does not exist" Oct 30 14:13:38.178884 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.178855 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk"] Oct 30 14:13:38.179394 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179374 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwkfh\" (UniqueName: \"kubernetes.io/projected/1990209a-3cb1-4793-9d68-ae869455f487-kube-api-access-gwkfh\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.179466 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179399 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b78d06-b249-4247-b0e7-78acefe36d87-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.179466 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179411 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.179466 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179420 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1990209a-3cb1-4793-9d68-ae869455f487-dshm\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.179466 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179436 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1990209a-3cb1-4793-9d68-ae869455f487-tls-certs\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.179466 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179445 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b78d06-b249-4247-b0e7-78acefe36d87-model-cache\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.179466 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.179452 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-685gd\" (UniqueName: \"kubernetes.io/projected/f4b78d06-b249-4247-b0e7-78acefe36d87-kube-api-access-685gd\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:38.182488 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.182463 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-w2-ec41e5-kserve-5fd775bd66-n4tgk"] Oct 30 14:13:38.214367 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.214289 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98"] Oct 30 14:13:38.218230 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.218184 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/premium-w2-ec41e5-kserve-6c5f6755dc-qrl98"] Oct 30 14:13:38.253239 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.253213 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1990209a-3cb1-4793-9d68-ae869455f487" path="/var/lib/kubelet/pods/1990209a-3cb1-4793-9d68-ae869455f487/volumes" Oct 30 14:13:38.253664 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:38.253650 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" path="/var/lib/kubelet/pods/f4b78d06-b249-4247-b0e7-78acefe36d87/volumes" Oct 30 14:13:40.815923 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:40.815888 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:44.919853 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:44.919813 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:45.375610 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:45.375564 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:45.517259 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:45.517222 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:46.008232 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:46.008189 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:47.116512 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:47.116471 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:13:50.531810 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:50.531775 5693 scope.go:122] "RemoveContainer" containerID="6a256bf9d3b8c8563ba30bbd635df26a75f21168c8ad4edf11d69c56e799a9e7" Oct 30 14:13:52.897179 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:52.897143 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:53.055582 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:53.055542 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:13:53.072044 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.072020 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:53.276618 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.276561 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:53.563055 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.563022 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 14:13:53.732283 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.732244 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h\" (UniqueName: \"kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h\") pod \"16d031a8-4853-490f-baf9-0e39d19ad754\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " Oct 30 14:13:53.732469 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.732327 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume\") pod \"16d031a8-4853-490f-baf9-0e39d19ad754\" (UID: \"16d031a8-4853-490f-baf9-0e39d19ad754\") " Oct 30 14:13:53.732721 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.732696 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume" pod "16d031a8-4853-490f-baf9-0e39d19ad754" (UID: "16d031a8-4853-490f-baf9-0e39d19ad754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:53.734405 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.734381 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h" pod "16d031a8-4853-490f-baf9-0e39d19ad754" (UID: "16d031a8-4853-490f-baf9-0e39d19ad754"). InnerVolumeSpecName "kube-api-access-8zv6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:53.833943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.833874 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zv6h\" (UniqueName: \"kubernetes.io/projected/16d031a8-4853-490f-baf9-0e39d19ad754-kube-api-access-8zv6h\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:53.833943 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:53.833903 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d031a8-4853-490f-baf9-0e39d19ad754-plugins-config-volume\") on node \"ip-10-0-130-43.ec2.internal\" DevicePath \"\"" Oct 30 14:13:54.198399 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.198309 5693 generic.go:356] "Generic (PLEG): container finished" podID="16d031a8-4853-490f-baf9-0e39d19ad754" containerID="71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1" exitCode=0 Oct 30 14:13:54.198399 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.198361 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" Oct 30 14:13:54.198904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.198394 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" event={"ID":"16d031a8-4853-490f-baf9-0e39d19ad754","Type":"ContainerDied","Data":"71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1"} Oct 30 14:13:54.198904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.198450 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b" event={"ID":"16d031a8-4853-490f-baf9-0e39d19ad754","Type":"ContainerDied","Data":"38d86f78ea8c80819ee3f84347972cc7b35b3547c0ca8a478d8460c4f3cbbeaa"} Oct 30 14:13:54.198904 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.198473 5693 scope.go:122] "RemoveContainer" containerID="71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1" Oct 30 14:13:54.207786 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.207764 5693 scope.go:122] "RemoveContainer" containerID="71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1" Oct 30 14:13:54.208052 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:13:54.208024 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1\": container with ID starting with 71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1 not found: ID does not exist" containerID="71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1" Oct 30 14:13:54.208116 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.208063 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1"} err="failed to get container status \"71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1\": rpc error: code = NotFound desc = could not find container \"71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1\": container with ID starting with 71f831476ead56d31ea304e422477bd4d68ed1c1c6dad3e881e3e0b4682eb2b1 not found: ID does not exist" Oct 30 14:13:54.255249 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.255220 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b"] Oct 30 14:13:54.259377 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:54.259357 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec41e5-ffc69748d-qzh2b"] Oct 30 14:13:56.252915 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:56.252871 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" path="/var/lib/kubelet/pods/16d031a8-4853-490f-baf9-0e39d19ad754/volumes" Oct 30 14:13:56.636035 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:56.635935 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:13:57.053444 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:57.053405 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:13:57.819173 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:57.819124 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:13:58.436725 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:58.436692 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:13:58.844168 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:58.844139 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:13:59.514742 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:59.514705 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:59.742067 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:59.742027 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:59.942799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:13:59.942701 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:01.021588 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:01.021551 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:02.360510 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:02.360474 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:02.495724 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:02.495692 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:07.122432 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:07.122393 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:10.047696 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:10.047656 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:14:10.487121 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:10.487093 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:14:11.246204 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:11.246170 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:14:11.867611 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:11.867570 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:14:12.252578 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:12.252547 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:14:12.776480 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:12.776435 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-666b98648-qjmq6_c3921b04-5eab-4fef-ad91-326b111dd118/maas-api/0.log" Oct 30 14:14:12.998053 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:12.998020 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-13946755-zsf8c_eddd73e0-4506-472b-aea2-be55a188a936/revoke-keys/0.log" Oct 30 14:14:15.977118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:15.977084 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-vmntf_9b500b45-7eed-428d-9bbd-28bc8ac1576d/postgres/0.log" Oct 30 14:14:19.866201 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:19.866111 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:20.345001 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:20.344965 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7998d67549-9b9rf_6757ec91-0fed-4966-a03d-ace6e941ed75/kube-auth-proxy/0.log" Oct 30 14:14:20.732640 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:20.732613 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-dhjq8_2a5295dc-10f1-476e-b269-54c956fb9490/istio-proxy/0.log" Oct 30 14:14:20.947866 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:20.947831 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-rjvw9_eed2fc6d-db5c-4423-9ae5-0334677b4e1f/istio-proxy/0.log" Oct 30 14:14:21.509106 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:21.509069 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8trj2_79896ce2-39e6-406c-9079-e5077b542ad3/payload-pre-processing/0.log" Oct 30 14:14:21.704001 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:21.703975 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-kg4wx_89c0aafb-9900-4eb3-90ec-d26f746bf51a/payload-processing/0.log" Oct 30 14:14:22.679048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:22.679002 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zb28c_80b106a2-9b74-4de0-8ea8-9b11914f9f61/main/0.log" Oct 30 14:14:22.869967 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:22.869937 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rct2s_dd44a7d4-7135-435f-9041-fc4f6fde603c/main/0.log" Oct 30 14:14:23.053915 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:14:23.053876 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:14:23.266789 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:23.266757 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-h4jxx_0cd02c9e-ae0b-4093-95a2-2ec4740b8055/main/0.log" Oct 30 14:14:23.663619 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:23.663561 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bct5z_0389eea9-e8d7-4d71-a2d2-378dc5ca820a/main/0.log" Oct 30 14:14:33.988436 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988338 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lc7xb/must-gather-x8q57"] Oct 30 14:14:33.988925 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988905 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988926 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988943 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988950 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988958 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988963 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d687829a-5d4b-469f-b1a3-262f612892a0" containerName="istio-proxy" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988978 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988983 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0512a21c-cfd6-47ec-9c02-4e2079109cc2" containerName="istio-proxy" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988992 5693 cpu_manager.go:414] "containerMap: removing container" podUID="eddd73e0-4506-472b-aea2-be55a188a936" containerName="revoke-keys" Oct 30 14:14:33.988999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.988998 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eddd73e0-4506-472b-aea2-be55a188a936" containerName="revoke-keys" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989017 5693 cpu_manager.go:414] "containerMap: removing container" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" containerName="payload-processing" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989024 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" containerName="payload-processing" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989031 5693 cpu_manager.go:414] "containerMap: removing container" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerName="payload-pre-processing" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989035 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerName="payload-pre-processing" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989045 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989049 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989108 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf039ca7-5fc7-4310-8c29-685bc70445d1" containerName="istio-proxy" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989117 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b78d06-b249-4247-b0e7-78acefe36d87" containerName="main" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989126 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04aea9d-1efd-4626-9bea-8449107bf8b0" containerName="istio-proxy" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989132 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1990209a-3cb1-4793-9d68-ae869455f487" containerName="main" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989140 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6770ae55-0287-4dc9-a718-01f0e2136e8c" containerName="istio-proxy" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989146 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="eddd73e0-4506-472b-aea2-be55a188a936" containerName="revoke-keys" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989153 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d031a8-4853-490f-baf9-0e39d19ad754" containerName="payload-processing" Oct 30 14:14:33.989349 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.989160 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f6e0a0-cb2c-4a60-92f7-a5c6d621ae65" containerName="payload-pre-processing" Oct 30 14:14:33.992395 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:33.992377 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.007493 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.007462 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lc7xb\"/\"kube-root-ca.crt\"" Oct 30 14:14:34.007778 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.007759 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lc7xb\"/\"default-dockercfg-g6nds\"" Oct 30 14:14:34.007855 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.007775 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lc7xb\"/\"openshift-service-ca.crt\"" Oct 30 14:14:34.021286 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.021260 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc7xb/must-gather-x8q57"] Oct 30 14:14:34.052623 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.052558 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:34.087402 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.087366 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fhsqk/must-gather-5q8xz"] Oct 30 14:14:34.091230 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.091211 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.096715 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.096689 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7nhw/must-gather-276k6"] Oct 30 14:14:34.100304 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.100283 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.108590 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.108568 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhsqk\"/\"openshift-service-ca.crt\"" Oct 30 14:14:34.108714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.108700 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fhsqk\"/\"default-dockercfg-mgtrq\"" Oct 30 14:14:34.111433 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.111410 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/must-gather-5q8xz"] Oct 30 14:14:34.112335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.112313 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfklp\" (UniqueName: \"kubernetes.io/projected/2e9424cd-e369-42e5-9b5f-57fc13d09e79-kube-api-access-bfklp\") pod \"must-gather-x8q57\" (UID: \"2e9424cd-e369-42e5-9b5f-57fc13d09e79\") " pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.112429 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.112378 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e9424cd-e369-42e5-9b5f-57fc13d09e79-must-gather-output\") pod \"must-gather-x8q57\" (UID: \"2e9424cd-e369-42e5-9b5f-57fc13d09e79\") " pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.120026 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.119998 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7nhw/must-gather-276k6"] Oct 30 14:14:34.197393 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.197360 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7nhw\"/\"kube-root-ca.crt\"" Oct 30 14:14:34.197393 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.197360 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7nhw\"/\"openshift-service-ca.crt\"" Oct 30 14:14:34.197630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.197612 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhsqk\"/\"kube-root-ca.crt\"" Oct 30 14:14:34.211324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.211296 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7nhw\"/\"default-dockercfg-5657q\"" Oct 30 14:14:34.213785 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.213758 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkmd\" (UniqueName: \"kubernetes.io/projected/c47dd34b-181f-4c50-8af5-19f92f903b8f-kube-api-access-sfkmd\") pod \"must-gather-276k6\" (UID: \"c47dd34b-181f-4c50-8af5-19f92f903b8f\") " pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.213927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.213810 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c47dd34b-181f-4c50-8af5-19f92f903b8f-must-gather-output\") pod \"must-gather-276k6\" (UID: \"c47dd34b-181f-4c50-8af5-19f92f903b8f\") " pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.213927 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.213919 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/859b8403-a9ca-440c-8aa3-c759a03e7b69-must-gather-output\") pod \"must-gather-5q8xz\" (UID: \"859b8403-a9ca-440c-8aa3-c759a03e7b69\") " pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.214027 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.213956 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfklp\" (UniqueName: \"kubernetes.io/projected/2e9424cd-e369-42e5-9b5f-57fc13d09e79-kube-api-access-bfklp\") pod \"must-gather-x8q57\" (UID: \"2e9424cd-e369-42e5-9b5f-57fc13d09e79\") " pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.214107 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.214080 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxxq\" (UniqueName: \"kubernetes.io/projected/859b8403-a9ca-440c-8aa3-c759a03e7b69-kube-api-access-6cxxq\") pod \"must-gather-5q8xz\" (UID: \"859b8403-a9ca-440c-8aa3-c759a03e7b69\") " pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.214155 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.214139 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e9424cd-e369-42e5-9b5f-57fc13d09e79-must-gather-output\") pod \"must-gather-x8q57\" (UID: \"2e9424cd-e369-42e5-9b5f-57fc13d09e79\") " pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.214487 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.214470 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e9424cd-e369-42e5-9b5f-57fc13d09e79-must-gather-output\") pod \"must-gather-x8q57\" (UID: \"2e9424cd-e369-42e5-9b5f-57fc13d09e79\") " pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.226999 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.226971 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:34.232655 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.232628 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfklp\" (UniqueName: \"kubernetes.io/projected/2e9424cd-e369-42e5-9b5f-57fc13d09e79-kube-api-access-bfklp\") pod \"must-gather-x8q57\" (UID: \"2e9424cd-e369-42e5-9b5f-57fc13d09e79\") " pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.314930 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.314843 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c47dd34b-181f-4c50-8af5-19f92f903b8f-must-gather-output\") pod \"must-gather-276k6\" (UID: \"c47dd34b-181f-4c50-8af5-19f92f903b8f\") " pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.314930 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.314894 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/859b8403-a9ca-440c-8aa3-c759a03e7b69-must-gather-output\") pod \"must-gather-5q8xz\" (UID: \"859b8403-a9ca-440c-8aa3-c759a03e7b69\") " pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.315119 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.314944 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxxq\" (UniqueName: \"kubernetes.io/projected/859b8403-a9ca-440c-8aa3-c759a03e7b69-kube-api-access-6cxxq\") pod \"must-gather-5q8xz\" (UID: \"859b8403-a9ca-440c-8aa3-c759a03e7b69\") " pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.315119 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.314985 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkmd\" (UniqueName: \"kubernetes.io/projected/c47dd34b-181f-4c50-8af5-19f92f903b8f-kube-api-access-sfkmd\") pod \"must-gather-276k6\" (UID: \"c47dd34b-181f-4c50-8af5-19f92f903b8f\") " pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.315288 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.315264 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c47dd34b-181f-4c50-8af5-19f92f903b8f-must-gather-output\") pod \"must-gather-276k6\" (UID: \"c47dd34b-181f-4c50-8af5-19f92f903b8f\") " pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.315335 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.315264 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/859b8403-a9ca-440c-8aa3-c759a03e7b69-must-gather-output\") pod \"must-gather-5q8xz\" (UID: \"859b8403-a9ca-440c-8aa3-c759a03e7b69\") " pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.316695 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.316680 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc7xb/must-gather-x8q57" Oct 30 14:14:34.334267 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.334235 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxxq\" (UniqueName: \"kubernetes.io/projected/859b8403-a9ca-440c-8aa3-c759a03e7b69-kube-api-access-6cxxq\") pod \"must-gather-5q8xz\" (UID: \"859b8403-a9ca-440c-8aa3-c759a03e7b69\") " pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.334705 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.334682 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkmd\" (UniqueName: \"kubernetes.io/projected/c47dd34b-181f-4c50-8af5-19f92f903b8f-kube-api-access-sfkmd\") pod \"must-gather-276k6\" (UID: \"c47dd34b-181f-4c50-8af5-19f92f903b8f\") " pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.406265 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.406227 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/must-gather-5q8xz" Oct 30 14:14:34.430675 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.430647 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7nhw/must-gather-276k6" Oct 30 14:14:34.614132 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.614045 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc7xb/must-gather-x8q57"] Oct 30 14:14:34.618034 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:14:34.617997 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e9424cd_e369_42e5_9b5f_57fc13d09e79.slice/crio-cb657d0406401661d6499db9c1567587c440a5b1744c69ce0edc60104b3fd6b3 WatchSource:0}: Error finding container cb657d0406401661d6499db9c1567587c440a5b1744c69ce0edc60104b3fd6b3: Status 404 returned error can't find the container with id cb657d0406401661d6499db9c1567587c440a5b1744c69ce0edc60104b3fd6b3 Oct 30 14:14:34.619826 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.619807 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:14:34.714401 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.714366 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/must-gather-5q8xz"] Oct 30 14:14:34.717711 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:14:34.717681 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod859b8403_a9ca_440c_8aa3_c759a03e7b69.slice/crio-c31d2061293e3f2c96dae48e9c514555dbc2eb2e458b156227a8464775299084 WatchSource:0}: Error finding container c31d2061293e3f2c96dae48e9c514555dbc2eb2e458b156227a8464775299084: Status 404 returned error can't find the container with id c31d2061293e3f2c96dae48e9c514555dbc2eb2e458b156227a8464775299084 Oct 30 14:14:34.766914 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:34.766880 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7nhw/must-gather-276k6"] Oct 30 14:14:34.782003 ip-10-0-130-43 kubenswrapper[5693]: W1030 14:14:34.781977 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc47dd34b_181f_4c50_8af5_19f92f903b8f.slice/crio-7a1ed6d79326bef92751f208e52fbbb74970a8ab3124541d9872911b348e55c7 WatchSource:0}: Error finding container 7a1ed6d79326bef92751f208e52fbbb74970a8ab3124541d9872911b348e55c7: Status 404 returned error can't find the container with id 7a1ed6d79326bef92751f208e52fbbb74970a8ab3124541d9872911b348e55c7 Oct 30 14:14:35.402341 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:35.402309 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/must-gather-5q8xz" event={"ID":"859b8403-a9ca-440c-8aa3-c759a03e7b69","Type":"ContainerStarted","Data":"c31d2061293e3f2c96dae48e9c514555dbc2eb2e458b156227a8464775299084"} Oct 30 14:14:35.403907 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:35.403878 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc7xb/must-gather-x8q57" event={"ID":"2e9424cd-e369-42e5-9b5f-57fc13d09e79","Type":"ContainerStarted","Data":"cb657d0406401661d6499db9c1567587c440a5b1744c69ce0edc60104b3fd6b3"} Oct 30 14:14:35.405090 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:35.405062 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7nhw/must-gather-276k6" event={"ID":"c47dd34b-181f-4c50-8af5-19f92f903b8f","Type":"ContainerStarted","Data":"7a1ed6d79326bef92751f208e52fbbb74970a8ab3124541d9872911b348e55c7"} Oct 30 14:14:36.416634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:36.415686 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/must-gather-5q8xz" event={"ID":"859b8403-a9ca-440c-8aa3-c759a03e7b69","Type":"ContainerStarted","Data":"04c6181ff9123e1c9ca1e3af6385d0119279a629385ec3fc8af474aacecb9d6a"} Oct 30 14:14:36.416634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:36.415741 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/must-gather-5q8xz" event={"ID":"859b8403-a9ca-440c-8aa3-c759a03e7b69","Type":"ContainerStarted","Data":"fb65892aa07fb469e87b87e408d99aaf96a795c04275430046e6f22ddddecbe2"} Oct 30 14:14:36.456024 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:36.455122 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fhsqk/must-gather-5q8xz" podStartSLOduration=1.7884966420000001 podStartE2EDuration="2.455103323s" totalImagesPullingTime="666.606681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:36.452032675 +0000 UTC m=+4252.318650851" watchObservedRunningTime="2026-10-30 14:14:36.455103323 +0000 UTC m=+4252.321721498" Oct 30 14:14:40.014871 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:40.014832 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kn246_d6aaafde-99f7-4cdb-9c95-381d18e5f1b1/global-pull-secret-syncer/0.log" Oct 30 14:14:40.893475 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:40.893438 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" Oct 30 14:14:41.304270 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:41.300663 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-43.ec2.internal_9c6cf73d04b1de3c177c6f664e9eb174/haproxy/0.log" Oct 30 14:14:50.682101 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:50.682059 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7nhw/must-gather-276k6" event={"ID":"c47dd34b-181f-4c50-8af5-19f92f903b8f","Type":"ContainerStarted","Data":"b05bb6b90e6d2c272b36aabcb9cc0807294ad3c2cdbfb0fb1e18973e1be4fd40"} Oct 30 14:14:50.682685 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:50.682656 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7nhw/must-gather-276k6" event={"ID":"c47dd34b-181f-4c50-8af5-19f92f903b8f","Type":"ContainerStarted","Data":"859133df467d119385686206465073c9cbba7ac2142590b9eb9827be7e2b8cd0"} Oct 30 14:14:50.728630 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:50.722623 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j7nhw/must-gather-276k6" podStartSLOduration=1.896857345 podStartE2EDuration="16.722606803s" totalImagesPullingTime="14.825749458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:50.721103283 +0000 UTC m=+4266.587721458" watchObservedRunningTime="2026-10-30 14:14:50.722606803 +0000 UTC m=+4266.589224977" Oct 30 14:14:51.464638 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.461768 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c"] Oct 30 14:14:51.473649 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.471986 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.488192 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.488137 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c"] Oct 30 14:14:51.650280 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.650239 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm6f\" (UniqueName: \"kubernetes.io/projected/a7697832-8260-4295-916d-8186643c7d8c-kube-api-access-hsm6f\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.650471 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.650318 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-lib-modules\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.650471 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.650402 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-podres\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.650644 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.650618 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-proc\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.650779 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.650668 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-sys\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.693815 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.693776 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc7xb/must-gather-x8q57" event={"ID":"2e9424cd-e369-42e5-9b5f-57fc13d09e79","Type":"ContainerStarted","Data":"29a27d64a5e280f3b6a24ea974fd0cb311f3df55de4328cc1baa12800ffa0b5b"} Oct 30 14:14:51.694396 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.694366 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc7xb/must-gather-x8q57" event={"ID":"2e9424cd-e369-42e5-9b5f-57fc13d09e79","Type":"ContainerStarted","Data":"80d746d2179c64aab8ebb926859fbc324bb81d08e2ea07f0dc7e17e1613b0e64"} Oct 30 14:14:51.732627 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.731638 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lc7xb/must-gather-x8q57" podStartSLOduration=2.642113201 podStartE2EDuration="18.73162128s" totalImagesPullingTime="16.089508079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:51.730646849 +0000 UTC m=+4267.597265024" watchObservedRunningTime="2026-10-30 14:14:51.73162128 +0000 UTC m=+4267.598239452" Oct 30 14:14:51.751332 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-sys\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.751536 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751366 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm6f\" (UniqueName: \"kubernetes.io/projected/a7697832-8260-4295-916d-8186643c7d8c-kube-api-access-hsm6f\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.751536 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751409 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-lib-modules\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.751536 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751427 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-podres\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.751536 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751514 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-proc\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.751812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751627 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-proc\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.751812 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.751673 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-sys\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.752021 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.752002 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-lib-modules\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.752088 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.752054 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7697832-8260-4295-916d-8186643c7d8c-podres\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.775624 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.774054 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm6f\" (UniqueName: \"kubernetes.io/projected/a7697832-8260-4295-916d-8186643c7d8c-kube-api-access-hsm6f\") pod \"perf-node-gather-daemonset-hdg9c\" (UID: \"a7697832-8260-4295-916d-8186643c7d8c\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:51.803622 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:51.799797 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:52.137670 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.135354 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/alertmanager/0.log" Oct 30 14:14:52.150626 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.143226 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c"] Oct 30 14:14:52.278101 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.278064 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/config-reloader/0.log" Oct 30 14:14:52.409844 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.409767 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/kube-rbac-proxy-web/0.log" Oct 30 14:14:52.532703 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.532666 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:14:52.539520 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.539496 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/kube-rbac-proxy/0.log" Oct 30 14:14:52.705678 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.704782 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" event={"ID":"a7697832-8260-4295-916d-8186643c7d8c","Type":"ContainerStarted","Data":"92b6f75a60967e80ae2bf7d63f621d3d878d02c0cf67ed50da4083542c69103b"} Oct 30 14:14:52.705678 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.704842 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" event={"ID":"a7697832-8260-4295-916d-8186643c7d8c","Type":"ContainerStarted","Data":"f11f7e0886d8833078e50ea6600adbdc73648de7becc3802f45fce734b2dbcb8"} Oct 30 14:14:52.705678 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.704891 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:52.710623 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.706841 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/kube-rbac-proxy-metric/0.log" Oct 30 14:14:52.731889 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.731857 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:14:52.839617 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.839357 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/prom-label-proxy/0.log" Oct 30 14:14:52.971827 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:52.970995 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b583b0bf-387f-46e7-8380-43fdedfe15e7/init-config-reloader/0.log" Oct 30 14:14:53.069789 ip-10-0-130-43 kubenswrapper[5693]: E1030 14:14:53.069729 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6zw7d" containerName="extractor" Oct 30 14:14:53.253649 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.253589 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t87s9_bdfaae46-aa7b-4311-8ea5-1f6716311a36/kube-state-metrics/0.log" Oct 30 14:14:53.377294 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.377252 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t87s9_bdfaae46-aa7b-4311-8ea5-1f6716311a36/kube-rbac-proxy-main/0.log" Oct 30 14:14:53.432482 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.432452 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:14:53.504455 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.504356 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t87s9_bdfaae46-aa7b-4311-8ea5-1f6716311a36/kube-rbac-proxy-self/0.log" Oct 30 14:14:53.534879 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.534845 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:14:53.640196 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.640165 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79cdc67ddb-bx2c7_187fc60f-7b2a-45dd-b5ff-13839099e07f/metrics-server/0.log" Oct 30 14:14:53.733242 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.733212 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:14:53.773634 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.773520 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-d5hqt_e85daeb3-1320-4a20-b831-0817f651f8e8/monitoring-plugin/0.log" Oct 30 14:14:53.970937 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:53.970894 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:14:54.528822 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:54.528766 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:14:54.726983 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:54.726244 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdws6_2ca8a3b3-0f1e-4bd2-8660-6d8add527517/node-exporter/0.log" Oct 30 14:14:54.865532 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:54.865454 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdws6_2ca8a3b3-0f1e-4bd2-8660-6d8add527517/kube-rbac-proxy/0.log" Oct 30 14:14:54.981649 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:54.981339 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:14:54.994887 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:54.994856 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdws6_2ca8a3b3-0f1e-4bd2-8660-6d8add527517/init-textfile/0.log" Oct 30 14:14:55.129977 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:55.129882 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mdb6n_6e86df5d-83ed-433e-9c0e-b5f59970dfe9/kube-rbac-proxy-main/0.log" Oct 30 14:14:55.262555 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:55.262522 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mdb6n_6e86df5d-83ed-433e-9c0e-b5f59970dfe9/kube-rbac-proxy-self/0.log" Oct 30 14:14:55.290714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:55.290679 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:14:55.396324 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:55.396245 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mdb6n_6e86df5d-83ed-433e-9c0e-b5f59970dfe9/openshift-state-metrics/0.log" Oct 30 14:14:56.177608 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.176291 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:14:56.457048 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.456960 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8vmwx_78a085e4-85e9-466c-984c-e08ccc0b8862/prometheus-operator/0.log" Oct 30 14:14:56.576720 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.576686 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8vmwx_78a085e4-85e9-466c-984c-e08ccc0b8862/kube-rbac-proxy/0.log" Oct 30 14:14:56.799889 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.799567 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:14:56.849002 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.845898 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-549bbd8b7d-lf666_a56d5807-0ea6-48b5-8487-64330925498a/telemeter-client/0.log" Oct 30 14:14:56.958080 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.958046 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:14:56.969799 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:56.969753 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-549bbd8b7d-lf666_a56d5807-0ea6-48b5-8487-64330925498a/reload/0.log" Oct 30 14:14:57.098311 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:57.098233 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-549bbd8b7d-lf666_a56d5807-0ea6-48b5-8487-64330925498a/kube-rbac-proxy/0.log" Oct 30 14:14:57.695029 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:57.694996 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:14:57.827098 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:57.827054 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:14:58.033625 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:58.030753 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:14:58.233880 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:58.233834 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:14:58.880714 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:58.880672 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:14:59.284672 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.284627 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:59.323684 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.323503 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:14:59.449347 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.449306 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:59.553581 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.553461 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:14:59.732118 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.732080 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" Oct 30 14:14:59.771841 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.771787 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-hdg9c" podStartSLOduration=8.771775008 podStartE2EDuration="8.771775008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:52.747841686 +0000 UTC m=+4268.614459860" watchObservedRunningTime="2026-10-30 14:14:59.771775008 +0000 UTC m=+4275.638393181" Oct 30 14:14:59.906641 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:14:59.905442 5693 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zncw_0aaa6be8-1e4d-44a0-91af-6a98718ba6e8/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:00.662645 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:00.661774 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:15:01.448633 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:01.448295 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:15:01.618644 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:01.617655 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:15:02.386848 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:02.386809 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:15:02.727900 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:02.727863 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-749c8d45cd-vlv69_ab22081e-2b0e-419c-8f42-d9beaef9607e/manager/0.log" Oct 30 14:15:02.732062 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:02.732030 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-545cc6f55f-j7g67_8174a28b-9f7a-4bde-942d-08e72cc5b097/console/0.log" Oct 30 14:15:02.898074 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:02.898037 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:15:02.926260 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:02.926219 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9xtc9_034a48e4-ba7c-4a2a-ae39-e64831996044/manager/0.log" Oct 30 14:15:03.877646 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:03.877589 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/0.log" Oct 30 14:15:04.272439 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:04.272400 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:15:04.360105 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:04.356334 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c7bfc749-57x8m_53c26842-64be-41e0-b367-57e23106974b/manager/1.log" Oct 30 14:15:05.591493 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:05.591398 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fhq4s_de229b37-17d3-4fbf-b0b9-5ffb01ff5cd1/proxy/0.log" Oct 30 14:15:07.726422 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:07.726387 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dg69d_da949e5b-eb60-4823-b417-41067e8d0959/dns/0.log" Oct 30 14:15:07.850005 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:07.849967 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dg69d_da949e5b-eb60-4823-b417-41067e8d0959/kube-rbac-proxy/0.log" Oct 30 14:15:08.379818 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:08.374638 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gd79z_fec11257-4c9d-48b9-9333-970360865ea7/dns-node-resolver/0.log" Oct 30 14:15:09.997315 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:09.997280 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kg5l_5276b8dc-f3cf-4d3b-a630-bd16aca9a373/node-ca/0.log" Oct 30 14:15:10.459874 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:10.459794 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7998d67549-9b9rf_6757ec91-0fed-4966-a03d-ace6e941ed75/kube-auth-proxy/0.log" Oct 30 14:15:10.600854 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:10.600818 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-dhjq8_2a5295dc-10f1-476e-b269-54c956fb9490/istio-proxy/0.log" Oct 30 14:15:10.666515 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:10.666483 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-rjvw9_eed2fc6d-db5c-4423-9ae5-0334677b4e1f/istio-proxy/0.log" Oct 30 14:15:10.863757 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:10.863715 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8trj2_79896ce2-39e6-406c-9079-e5077b542ad3/payload-pre-processing/0.log" Oct 30 14:15:10.933529 ip-10-0-130-43 kubenswrapper[5693]: I1030 14:15:10.933376 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-kg4wx_89c0aafb-9900-4eb3-90ec-d26f746bf51a/payload-processing/0.log"